Oct 30 21:41:41.671245 ip-10-0-141-104 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:41:42.037928 ip-10-0-141-104 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:41:42.037928 ip-10-0-141-104 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:41:42.037928 ip-10-0-141-104 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:41:42.037928 ip-10-0-141-104 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:41:42.041546 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041474 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:42.041546 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041545 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:42.041546 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041549 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041554 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041556 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041559 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041562 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041565 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041567 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041570 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041574 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041581 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041585 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041588 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041590 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041593 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041596 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041600 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041604 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041607 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041610 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:42.041634 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041612 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041615 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041617 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041620 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041622 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041624 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041627 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041630 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041632 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041635 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041638 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041645 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041648 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041651 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041653 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041656 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041658 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041661 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041664 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041666 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:42.042082 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041669 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041672 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041674 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041677 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041679 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041682 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041684 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041686 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041689 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041691 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041694 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041696 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041698 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041701 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041704 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041707 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041709 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041712 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041714 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:42.042550 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041717 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041719 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041722 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041724 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041726 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041729 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041732 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041734 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041737 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041739 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041741 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041744 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041746 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041749 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041751 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041754 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041757 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041759 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041761 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041764 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:42.043014 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041766 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041768 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041771 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041773 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041776 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041778 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041781 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041784 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041786 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041789 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041792 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041794 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041797 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041799 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041801 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041804 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041806 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041809 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041811 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:42.043695 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041814 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041817 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041819 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041822 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041824 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041826 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.041829 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042577 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042584 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042588 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042591 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042593 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042596 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042599 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042601 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042604 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042606 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042608 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042611 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042615 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:42.044275 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042618 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042621 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042624 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042626 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042629 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042631 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042634 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042637 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042639 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042643 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042645 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042649 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042653 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042656 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042659 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042661 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042664 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042666 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:42.044766 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042669 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042671 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042674 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042676 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042679 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042682 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042684 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042687 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042689 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042692 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042696 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042699 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042702 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042705 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042708 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042711 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042713 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042716 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042719 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:42.045346 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042721 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042724 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042726 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042729 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042731 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042733 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042736 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042738 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042741 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042744 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042746 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042749 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042751 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042753 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042756 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042759 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042761 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042763 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042766 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042769 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:42.045860 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042771 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042774 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042777 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042780 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042782 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042785 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042787 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042790 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042792 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042795 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042797 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042799 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042802 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042804 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042807 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042809 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042811 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042814 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042816 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042819 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:42.046339 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042821 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042824 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042826 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042829 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042831 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042834 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042836 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042838 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042841 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042843 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042846 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042849 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042852 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042856 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042858 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.042861 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045112 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045123 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045126 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045129 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:42.046799 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045132 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045135 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045138 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045141 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045144 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045146 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045149 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045152 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045155 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045157 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045160 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045163 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045165 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045168 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045170 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045173 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045175 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045178 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045180 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:42.047274 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045183 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045185 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045188 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045190 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045193 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045196 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045199 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045201 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045204 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045207 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045210 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045212 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045215 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045219 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045223 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045226 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045229 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045231 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045234 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:42.047703 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045238 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045242 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045244 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045248 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045251 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045253 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045256 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045258 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045261 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045263 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045266 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045269 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045271 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045274 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045276 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045279 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045281 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045284 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045286 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:42.048152 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045288 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045291 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045294 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045297 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045299 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045303 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045306 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045308 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045311 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045314 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045316 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045319 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045322 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045324 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045326 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045329 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045331 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045334 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045337 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045339 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:42.048584 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045342 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045344 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045347 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045350 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045353 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045355 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045358 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045360 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045363 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045366 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045368 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045371 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045373 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045376 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045379 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045381 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045384 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045386 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045389 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:42.049044 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045392 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045394 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045397 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045400 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045402 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.045406 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046257 2581 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046266 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046272 2581 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046277 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046281 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046285 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046289 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046293 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046296 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046299 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046304 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046307 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046310 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046313 2581 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046316 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046319 2581 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046322 2581 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:41:42.049483 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046324 2581 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046329 2581 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046331 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046334 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046338 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046341 2581 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046346 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046349 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046352 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046356 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046359 2581 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046361 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046364 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046367 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046370 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046374 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046377 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046381 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046383 2581 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046386 2581 flags.go:64] FLAG: --enable-server="true" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046389 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046393 2581 flags.go:64] FLAG: --event-burst="100" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046396 2581 flags.go:64] FLAG: --event-qps="50" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046399 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:41:42.050010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046402 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046404 2581 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046409 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046412 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046415 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046418 2581 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046421 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046423 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046426 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046429 2581 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046432 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046435 2581 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046437 2581 flags.go:64] FLAG: --feature-gates="" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046442 2581 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046445 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046447 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046451 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046454 2581 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046456 2581 flags.go:64] FLAG: --help="false" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046459 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046462 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046465 2581 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046468 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:41:42.050634 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046471 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046474 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046477 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046480 2581 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046482 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046485 2581 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046488 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046490 2581 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046493 2581 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046496 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046499 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046501 2581 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046504 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046507 2581 flags.go:64] FLAG: --lock-file="" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046510 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046512 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046515 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046520 2581 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046523 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046526 2581 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046528 2581 flags.go:64] FLAG: --logging-format="text" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046531 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046534 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046537 2581 flags.go:64] FLAG: --manifest-url="" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046540 2581 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:41:42.051180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046545 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046548 2581 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046551 2581 flags.go:64] FLAG: --max-pods="110" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046554 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046557 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046560 2581 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046563 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046566 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046569 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046572 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046579 2581 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046583 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046585 2581 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046588 2581 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046591 2581 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046594 2581 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046596 2581 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046599 2581 flags.go:64] FLAG: --port="10250" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046602 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046604 2581 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-08f140ac732cf41ed" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046608 2581 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046611 2581 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046614 2581 flags.go:64] FLAG: --register-node="true" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046617 2581 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046620 2581 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046623 2581 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:41:42.051755 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046626 2581 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046628 2581 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046632 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046635 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046638 2581 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046641 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046644 2581 flags.go:64] FLAG: --runonce="false" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046647 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046650 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046652 2581 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046655 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046658 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046661 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046664 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046667 2581 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046669 2581 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046672 2581 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046675 2581 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046678 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046680 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046684 2581 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046687 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046692 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046694 2581 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046697 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:41:42.052387 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046701 2581 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046704 2581 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046707 2581 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046710 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046713 2581 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046716 2581 flags.go:64] FLAG: --v="2" Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046720 2581 flags.go:64] FLAG: --version="false" Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046724 2581 flags.go:64] FLAG: --vmodule="" Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046727 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.046730 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046856 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046859 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046862 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046864 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046867 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046869 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046872 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046874 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046877 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046879 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046882 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:42.053072 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046884 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046887 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046890 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046892 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046895 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046897 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046900 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046902 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046905 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046907 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046909 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046915 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046918 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046922 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046925 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046927 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046930 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046933 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046935 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046938 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:42.053562 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046953 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046957 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046959 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046962 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046964 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046967 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046969 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046972 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046974 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046977 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046980 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046982 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046985 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046987 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046989 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046992 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046996 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.046999 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047002 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:42.054038 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047004 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047007 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047009 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047012 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047016 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047018 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047022 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047024 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047028 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047031 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047033 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047036 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047038 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047041 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047043 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047046 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047048 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047051 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047053 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:42.054501 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047056 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047058 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047060 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047063 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047066 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047068 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047071 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047073 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047076 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047078 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047080 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047083 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047085 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047088 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047090 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047092 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047095 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047098 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047101 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:42.054928 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047105 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047108 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047112 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047115 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047118 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047120 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047123 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047125 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047128 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047130 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047133 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047135 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047138 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047140 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047143 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047145 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047148 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:42.055372 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047150 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.047176 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047208 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047213 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047216 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047219 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047221 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047224 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047227 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047229 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047232 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047234 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047236 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047239 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047242 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047246 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:42.055761 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047249 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047251 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047254 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047257 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047260 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047263 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047266 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047268 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047271 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047273 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047276 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047279 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047284 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047287 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047289 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047292 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047294 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047297 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047299 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:42.056153 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047302 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047304 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047307 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047309 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047312 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047315 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047317 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047320 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047322 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047325 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047327 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047331 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047335 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047337 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047339 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047342 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047344 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047347 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047350 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047352 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:42.056593 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047354 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047358 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047361 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047363 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047366 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047368 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047371 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047373 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047376 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047378 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047380 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047383 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047385 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047388 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047391 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047395 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047398 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047401 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047404 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047406 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:42.057060 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047409 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047412 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047414 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047418 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047422 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047425 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047427 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047430 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047432 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047435 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047438 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047441 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047443 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047446 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047448 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047451 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047453 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047456 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047458 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:42.057515 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047461 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047463 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047465 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047468 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047471 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047473 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047476 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047478 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047482 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047485 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047487 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047490 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047492 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.047494 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.053897 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.053912 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.053928 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.053932 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054002 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054007 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:42.058048 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054010 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054013 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054016 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054018 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054021 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054024 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054027 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054029 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054031 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054034 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054037 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054039 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054042 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054045 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054048 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054050 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054053 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054055 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054057 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:42.058507 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054060 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054063 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054065 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054068 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054070 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054073 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054075 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054078 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054082 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054084 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054087 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054091 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054095 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054098 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054101 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054104 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054106 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054109 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054111 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054114 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:42.058938 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054116 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054119 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054122 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054124 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054127 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054129 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054132 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054135 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054137 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054140 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054142 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054145 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054147 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054150 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054152 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054155 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054157 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054160 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054163 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:42.059405 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054165 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054168 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054171 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054174 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054177 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054179 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054182 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054184 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054187 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054189 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054192 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054194 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054197 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054199 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054202 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054204 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054207 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054209 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054212 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054215 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:42.059911 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054217 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054220 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054223 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054225 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054227 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054230 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054232 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054235 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054237 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054240 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054242 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054245 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054247 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054250 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054253 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054258 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054260 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054263 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054265 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:42.060373 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054268 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054270 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054273 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054275 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054277 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054280 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054282 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.054308 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054438 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054442 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054445 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054448 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054450 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054453 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054455 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054458 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:42.060818 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054460 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054463 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054465 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054468 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054470 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054473 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054475 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054477 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054480 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054482 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054485 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054487 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054490 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054493 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054495 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054498 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054500 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054503 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054506 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054508 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:42.061195 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054511 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054514 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054516 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054519 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054522 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054524 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054526 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054529 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054531 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054534 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054536 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054538 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054541 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054543 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054546 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054548 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054550 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054553 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054555 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054557 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:42.061664 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054560 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054562 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054565 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054567 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054570 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054572 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054575 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054578 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054580 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054583 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054585 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054588 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054590 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054593 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054596 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054598 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054601 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054603 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054606 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:42.062142 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054610 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054614 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054616 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054619 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054621 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054624 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054626 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054629 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054631 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054633 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054636 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054638 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054641 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054643 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054646 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054648 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054651 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054653 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054656 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:42.062569 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054658 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054661 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054665 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054669 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054672 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054675 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054678 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054680 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054682 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054685 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054687 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054690 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054692 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054694 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054697 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054699 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054701 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054704 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054706 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:42.063094 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054709 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.054731 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054766 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054770 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054773 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054776 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054780 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054783 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054786 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054789 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054792 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054794 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054796 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054799 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054801 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:42.063540 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054804 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054806 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054809 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054812 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054814 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054817 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054819 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054821 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054824 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054826 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054829 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054831 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054834 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054837 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054840 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054843 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054846 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054849 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054851 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:42.063976 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054854 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054856 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054859 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054861 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054864 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054866 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054868 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054871 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054873 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054876 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054878 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054881 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054883 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054886 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054889 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054892 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054894 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054896 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054899 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:42.064513 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054901 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054903 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054906 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054908 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054911 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054913 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054915 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054918 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054920 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054923 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054925 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054930 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054933 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054935 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054938 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054954 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054957 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054960 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054963 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054965 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:42.064975 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054968 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054970 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054973 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054976 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054978 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054981 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054983 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054985 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054989 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054992 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054994 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054996 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.054999 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055001 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055004 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055006 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055009 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055011 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055013 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055016 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:42.065518 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055018 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055020 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055023 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055027 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055029 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055032 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055034 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055037 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055039 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055042 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055044 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055048 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055050 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055053 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:42.055055 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.055107 2581 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.057101 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.057874 2581 server.go:1014] "Starting client certificate rotation" Oct 30 21:41:42.066078 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.058424 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:41:42.066596 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.059012 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:41:42.066596 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.059839 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:41:42.066596 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.060905 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:41:42.071010 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.070998 2581 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:41:42.074488 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.074475 2581 log.go:25] "Validated CRI v1 image API" Oct 30 21:41:42.075203 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.075181 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:41:42.079927 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.079905 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 da621e9c-a212-44de-8106-3bd3ab5cb51f:/dev/nvme0n1p3 dbe0243e-a3dc-4862-817c-ff7c58b21379:/dev/nvme0n1p4] Oct 30 21:41:42.080016 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.079926 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:41:42.084878 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.084765 2581 manager.go:217] Machine: {Timestamp:2026-10-30 21:41:42.08345789 +0000 UTC m=+0.300423700 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3196622 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec283ca0bea5853aeeef5080fba298b2 SystemUUID:ec283ca0-bea5-853a-eeef-5080fba298b2 BootID:404ce9a8-348c-423c-9019-6dec705796a8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:19:4e:ba:b0:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:19:4e:ba:b0:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:49:c3:bf:88:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:41:42.084878 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.084867 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:41:42.085069 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.084977 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:41:42.089038 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.089017 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:41:42.089188 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.089040 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-104.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:41:42.089273 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.089192 2581 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:41:42.089273 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.089203 2581 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:41:42.089273 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.089250 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:41:42.090172 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.090160 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:41:42.091320 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.091309 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:41:42.095508 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.095495 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:41:42.095632 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.095621 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:41:42.096434 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.096423 2581 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:41:42.096486 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.096440 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:41:42.096486 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.096453 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:41:42.096486 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.096467 2581 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:41:42.096486 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.096477 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:41:42.096801 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.096787 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:41:42.099108 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.099095 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:41:42.105430 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.105417 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:41:42.106364 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.106352 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:41:42.106402 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.106381 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:41:42.106532 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.106522 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:41:42.106562 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.106538 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:41:42.106562 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.106546 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:41:42.106562 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.106551 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:41:42.106562 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.106557 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:41:42.106562 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.106562 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:41:42.107320 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.107311 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:41:42.107348 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.107320 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:41:42.107348 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.107327 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:41:42.107348 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.107333 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:41:42.107348 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.107346 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:41:42.107542 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.107355 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:41:42.108867 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.108859 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:41:42.108867 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.108868 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:41:42.112507 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.112487 2581 server.go:1263] "Started kubelet" Oct 30 21:41:42.112664 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.112544 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:41:42.112718 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.112602 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:41:42.112778 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.112766 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:41:42.113233 ip-10-0-141-104 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:41:42.114697 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.114198 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:41:42.115590 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.115571 2581 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:41:42.119429 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.119412 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:41:42.120897 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.120872 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:41:42.122066 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.122048 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:41:42.122202 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.122148 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-104.us-west-1.compute.internal\" not found" Oct 30 21:41:42.122750 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.122732 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:41:42.122750 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.122734 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:41:42.122867 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.122763 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:41:42.122867 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.122789 2581 factory.go:55] Registering systemd factory Oct 30 21:41:42.122867 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.122843 2581 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:41:42.122867 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.122853 2581 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:41:42.123018 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.122863 2581 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:41:42.123514 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.123499 2581 factory.go:153] Registering CRI-O factory Oct 30 21:41:42.123636 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.123625 2581 factory.go:223] Registration of the crio container factory successfully Oct 30 21:41:42.123752 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.123742 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:41:42.123853 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.123842 2581 factory.go:103] Registering Raw factory Oct 30 21:41:42.123955 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.123933 2581 manager.go:1199] Started watching for new ooms in manager Oct 30 21:41:42.125143 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.125123 2581 manager.go:319] Starting recovery of all containers Oct 30 21:41:42.134598 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.134567 2581 manager.go:324] Recovery completed Oct 30 21:41:42.139465 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.139452 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:42.142754 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.142722 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:42.142835 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.142769 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:42.142835 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.142784 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:42.143367 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.143353 2581 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:41:42.143367 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.143366 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:42.143481 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.143366 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:41:42.143481 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.143432 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:41:42.144099 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.144082 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:42.144163 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.144108 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:42.144163 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.144120 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:42.145588 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.145576 2581 policy_none.go:50] "Start" Oct 30 21:41:42.145629 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.145590 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:41:42.145629 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.145604 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:41:42.147521 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.147505 2581 policy_none.go:44] "Start" Oct 30 21:41:42.174665 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.174637 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:41:42.177616 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.177601 2581 manager.go:341] "Starting Device Plugin manager" Oct 30 21:41:42.177680 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.177631 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:41:42.177680 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.177643 2581 server.go:87] "Starting device plugin registration server" Oct 30 21:41:42.177866 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.177854 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:41:42.177900 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.177870 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:41:42.178068 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.178054 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:41:42.178173 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.178126 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:41:42.178173 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.178135 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:41:42.178484 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.178467 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:41:42.178556 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.178515 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-104.us-west-1.compute.internal\" not found" Oct 30 21:41:42.256162 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.256128 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-104.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:41:42.259200 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.256134 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-104.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:41:42.259806 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.256325 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-104.us-west-1.compute.internal.18e36c9de9d75b89 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-104.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-104.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:42.112459657 +0000 UTC m=+0.329425468,LastTimestamp:2026-10-30 21:41:42.112459657 +0000 UTC m=+0.329425468,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-104.us-west-1.compute.internal,}" Oct 30 21:41:42.263919 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.263896 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:41:42.265116 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.265101 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:41:42.265173 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.265124 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:41:42.265173 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.265148 2581 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:41:42.265258 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.265186 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:41:42.278688 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.278669 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:42.279846 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.279831 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:42.279919 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.279858 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:42.279919 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.279868 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:42.279919 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.279889 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.334027 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.333840 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:41:42.334027 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.333918 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5wwb5" Oct 30 21:41:42.366355 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.366325 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal"] Oct 30 21:41:42.366445 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.366387 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:42.367265 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.367249 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:42.367342 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.367277 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:42.367342 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.367288 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:42.369429 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.369413 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:42.369564 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.369550 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.369618 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.369584 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:42.371226 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.371209 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:42.371284 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.371236 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:42.371284 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.371245 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:42.371342 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.371209 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:42.371342 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.371329 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:42.371342 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.371342 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:42.373372 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.373354 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.373453 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.373385 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:42.374093 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.374078 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:42.374176 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.374103 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:42.374176 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.374116 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:42.407184 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.407164 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-104.us-west-1.compute.internal\" not found" node="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.409531 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.409517 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-104.us-west-1.compute.internal\" not found" node="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.418673 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.418655 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:42.424028 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.424012 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff30ba81bde53b1c4da7b24cd2c2a4b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"ff30ba81bde53b1c4da7b24cd2c2a4b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.424109 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.424042 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.424109 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.424067 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.471073 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.471056 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:42.520476 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.520455 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:42.521628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.521613 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:42.521986 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.521972 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.522023 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:42.521993 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-104.us-west-1.compute.internal\": node \"ip-10-0-141-104.us-west-1.compute.internal\" not found" Oct 30 21:41:42.525044 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.525027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff30ba81bde53b1c4da7b24cd2c2a4b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"ff30ba81bde53b1c4da7b24cd2c2a4b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.525114 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.525052 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.525114 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.525067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.525114 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.525093 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.525205 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.525117 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff30ba81bde53b1c4da7b24cd2c2a4b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"ff30ba81bde53b1c4da7b24cd2c2a4b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.525205 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.525123 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.623224 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.623175 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.710185 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.710161 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.712333 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.712317 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.740080 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.740063 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:41:42.740161 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.740151 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:41:42.797359 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:42.797337 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:41:43.058048 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.057972 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:41:43.058557 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.058143 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:41:43.058557 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.058157 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:41:43.058557 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.058154 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:41:43.058557 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.058228 2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3ee7570faedf4a5ca3084ec2df71db2-8c14c679226562c9.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-104.us-west-1.compute.internal.18e36c9deba628c0\": read tcp 10.0.141.104:54492->77.112.223.82:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-104.us-west-1.compute.internal.18e36c9deba628c0 default 4844 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-104.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-104.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-104.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:42 +0000 UTC,LastTimestamp:2026-10-30 21:41:42.279872421 +0000 UTC m=+0.496838230,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-104.us-west-1.compute.internal,}" Oct 30 21:41:43.096864 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.096842 2581 apiserver.go:52] "Watching apiserver" Oct 30 21:41:43.121817 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.121797 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:41:43.338791 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.338706 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:42 +0000 UTC" deadline="2028-07-10 10:57:42.958999456 +0000 UTC" Oct 30 21:41:43.338791 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.338741 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14845h15m59.620261875s" Oct 30 21:41:43.374639 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.374613 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:41:43.374987 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.374953 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-jnj6f","openshift-ovn-kubernetes/ovnkube-node-wgpl9","openshift-dns/node-resolver-5xm5p","openshift-image-registry/node-ca-fgq49","openshift-multus/multus-additional-cni-plugins-5g5tc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-vbnk5","openshift-network-operator/iptables-alerter-xlpzs","kube-system/global-pull-secret-syncer-z258k","kube-system/konnectivity-agent-lhsfr","kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2","openshift-multus/multus-n5xj4","openshift-cluster-node-tuning-operator/tuned-qthxc"] Oct 30 21:41:43.377531 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.377516 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:43.377621 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.377603 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:41:43.379714 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.379700 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.383574 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.383550 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.385794 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.385776 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.388113 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.388097 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.390261 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.390248 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:43.390315 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.390298 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:41:43.392540 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.392522 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.394804 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.394789 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:43.394880 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.394836 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:41:43.397034 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.397015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:43.399365 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.399350 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.401674 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.401659 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.403942 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.403923 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.424121 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.424102 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:41:43.429399 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429379 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-tuned\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.429494 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429408 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-ovn\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.429494 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429445 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-netns\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.429494 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:43.429660 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429500 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-registration-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.429660 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429518 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-socket-dir-parent\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.429660 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429533 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysconfig\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.429660 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429553 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-host\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.429660 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429626 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgn5s\" (UniqueName: \"kubernetes.io/projected/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kube-api-access-lgn5s\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.429916 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-multus-certs\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.429916 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:43.429916 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429729 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-var-lib-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.429916 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429769 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-log-socket\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.429916 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429791 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.429916 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429807 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:43.429916 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429845 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-sys-fs\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.429916 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429859 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.429916 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-etc-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.429916 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429888 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/641871f0-a0d5-4be1-b2d0-3c7603211724-hosts-file\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.429975 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5200dd70-ca15-4547-b344-c772e014e92b-iptables-alerter-script\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430010 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-systemd\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-bin\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430102 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-netd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40cea1c9-e71c-4451-926c-398f36e9691c-konnectivity-ca\") pod \"konnectivity-agent-lhsfr\" (UID: \"40cea1c9-e71c-4451-926c-398f36e9691c\") " pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430154 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-conf-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430198 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5200dd70-ca15-4547-b344-c772e014e92b-host-slash\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqcc\" (UniqueName: \"kubernetes.io/projected/61f77d06-9723-4c98-8dd8-80c01d978e2f-kube-api-access-zgqcc\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cnibin\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430332 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-binary-copy\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.430411 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430386 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-systemd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430431 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430469 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-cnibin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430501 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-multus\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430537 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfw6m\" (UniqueName: \"kubernetes.io/projected/a749e90b-1748-48e1-b809-89cece3bab0a-kube-api-access-zfw6m\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430558 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-run\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430613 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-socket-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430707 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-device-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-kubernetes\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430768 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-script-lib\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430822 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-system-cni-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430870 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-netns\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430895 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-node-log\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430911 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40cea1c9-e71c-4451-926c-398f36e9691c-agent-certs\") pod \"konnectivity-agent-lhsfr\" (UID: \"40cea1c9-e71c-4451-926c-398f36e9691c\") " pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:43.430982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.430940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-slash\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431006 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-kubelet\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431022 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-sys\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431038 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-lib-modules\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431155 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a85ea4e-16f2-4088-ba73-7e996f46e151-serviceca\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431183 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431211 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-k8s-cni-cncf-io\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-bin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431284 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431299 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-var-lib-kubelet\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431320 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641871f0-a0d5-4be1-b2d0-3c7603211724-tmp-dir\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431350 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a85ea4e-16f2-4088-ba73-7e996f46e151-host\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-system-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-etc-kubernetes\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431420 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-config\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.431470 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431436 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-env-overrides\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431460 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg55q\" (UniqueName: \"kubernetes.io/projected/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-kube-api-access-dg55q\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-kubelet-config\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-dbus\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-cni-binary-copy\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8lr5\" (UniqueName: \"kubernetes.io/projected/5200dd70-ca15-4547-b344-c772e014e92b-kube-api-access-q8lr5\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431693 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-modprobe-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431722 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d29a054-4a50-4709-81c8-01c50df2944e-ovn-node-metrics-cert\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431739 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-systemd-units\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431766 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-hostroot\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431801 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-daemon-config\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431824 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gq6m\" (UniqueName: \"kubernetes.io/projected/1a631286-ec63-4b0a-8fc6-f97280e181a1-kube-api-access-7gq6m\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431869 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-conf\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431898 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-tmp\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.431950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431924 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54q8h\" (UniqueName: \"kubernetes.io/projected/1d29a054-4a50-4709-81c8-01c50df2944e-kube-api-access-54q8h\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.432439 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.431962 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqm4k\" (UniqueName: \"kubernetes.io/projected/641871f0-a0d5-4be1-b2d0-3c7603211724-kube-api-access-gqm4k\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.432439 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.432002 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-kubelet\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.432439 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.432048 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6p77\" (UniqueName: \"kubernetes.io/projected/3a85ea4e-16f2-4088-ba73-7e996f46e151-kube-api-access-z6p77\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.435443 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.435424 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:41:43.435619 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.435605 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:41:43.435823 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.435801 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9qhlv\"" Oct 30 21:41:43.435823 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.435816 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:41:43.436034 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.436007 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:41:43.436094 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.436055 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:41:43.436161 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.436147 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:41:43.439106 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.439088 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:41:43.439184 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.439122 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gscd\"" Oct 30 21:41:43.439184 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.439175 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:41:43.441080 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.441064 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:41:43.441151 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.441107 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:41:43.441188 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.441152 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:41:43.441222 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.441193 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-82rrg\"" Oct 30 21:41:43.443467 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.443451 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:41:43.443532 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.443479 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:41:43.532822 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.532802 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-host\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.532966 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.532830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgn5s\" (UniqueName: \"kubernetes.io/projected/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kube-api-access-lgn5s\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.532966 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.532848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-multus-certs\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.532966 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.532931 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:43.533148 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.532973 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-host\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.533148 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.532987 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-var-lib-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.533148 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-log-socket\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.533148 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533058 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-multus-certs\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.533148 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.533148 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533102 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:43.533148 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533115 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-log-socket\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.533148 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533131 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-sys-fs\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.533148 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533130 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-var-lib-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533171 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533187 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-sys-fs\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.533219 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533230 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-etc-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/641871f0-a0d5-4be1-b2d0-3c7603211724-hosts-file\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533285 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-etc-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.533291 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret podName:70fbe0ee-70e4-47b1-8094-4faa96292c6b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:44.033250409 +0000 UTC m=+2.250216228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret") pod "global-pull-secret-syncer-z258k" (UID: "70fbe0ee-70e4-47b1-8094-4faa96292c6b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533295 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/641871f0-a0d5-4be1-b2d0-3c7603211724-hosts-file\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533339 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5200dd70-ca15-4547-b344-c772e014e92b-iptables-alerter-script\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533355 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533382 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-systemd\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-bin\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533424 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-netd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40cea1c9-e71c-4451-926c-398f36e9691c-konnectivity-ca\") pod \"konnectivity-agent-lhsfr\" (UID: \"40cea1c9-e71c-4451-926c-398f36e9691c\") " pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:43.533561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533446 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-systemd\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533464 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-conf-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533479 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533499 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-netd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533507 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-conf-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5200dd70-ca15-4547-b344-c772e014e92b-host-slash\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533525 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-bin\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533540 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqcc\" (UniqueName: \"kubernetes.io/projected/61f77d06-9723-4c98-8dd8-80c01d978e2f-kube-api-access-zgqcc\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cnibin\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533596 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5200dd70-ca15-4547-b344-c772e014e92b-host-slash\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533604 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-binary-copy\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.533989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534008 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cnibin\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-systemd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534132 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-systemd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-cnibin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.534395 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-multus\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534197 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfw6m\" (UniqueName: \"kubernetes.io/projected/a749e90b-1748-48e1-b809-89cece3bab0a-kube-api-access-zfw6m\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534245 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-cnibin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534266 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-multus\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534266 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-run\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534327 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-run\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-socket-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-device-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534557 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534582 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-kubernetes\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534617 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-script-lib\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534653 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-system-cni-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534687 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-netns\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534691 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-socket-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534748 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-netns\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.535233 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-node-log\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534820 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40cea1c9-e71c-4451-926c-398f36e9691c-agent-certs\") pod \"konnectivity-agent-lhsfr\" (UID: \"40cea1c9-e71c-4451-926c-398f36e9691c\") " pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534826 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-kubernetes\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534859 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-slash\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534891 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-device-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-kubelet\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534940 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-kubelet\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-sys\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534983 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-sys\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.534983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-lib-modules\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a85ea4e-16f2-4088-ba73-7e996f46e151-serviceca\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535142 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-k8s-cni-cncf-io\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-bin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535227 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-var-lib-kubelet\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535273 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641871f0-a0d5-4be1-b2d0-3c7603211724-tmp-dir\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.536049 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a85ea4e-16f2-4088-ba73-7e996f46e151-host\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-system-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535383 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-etc-kubernetes\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-config\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535446 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-env-overrides\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535478 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg55q\" (UniqueName: \"kubernetes.io/projected/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-kube-api-access-dg55q\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-kubelet-config\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-dbus\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535596 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-cni-binary-copy\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8lr5\" (UniqueName: \"kubernetes.io/projected/5200dd70-ca15-4547-b344-c772e014e92b-kube-api-access-q8lr5\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535667 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-modprobe-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d29a054-4a50-4709-81c8-01c50df2944e-ovn-node-metrics-cert\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-systemd-units\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-hostroot\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535782 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-script-lib\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-daemon-config\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.536914 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq6m\" (UniqueName: \"kubernetes.io/projected/1a631286-ec63-4b0a-8fc6-f97280e181a1-kube-api-access-7gq6m\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535871 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-conf\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535901 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-tmp\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535110 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-lib-modules\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535949 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54q8h\" (UniqueName: \"kubernetes.io/projected/1d29a054-4a50-4709-81c8-01c50df2944e-kube-api-access-54q8h\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535979 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-k8s-cni-cncf-io\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.535986 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqm4k\" (UniqueName: \"kubernetes.io/projected/641871f0-a0d5-4be1-b2d0-3c7603211724-kube-api-access-gqm4k\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-kubelet\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536084 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6p77\" (UniqueName: \"kubernetes.io/projected/3a85ea4e-16f2-4088-ba73-7e996f46e151-kube-api-access-z6p77\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536115 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-tuned\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536144 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-ovn\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536178 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-netns\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536236 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-registration-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-socket-dir-parent\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysconfig\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.537628 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536426 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysconfig\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536503 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-slash\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536548 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-node-log\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536659 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536716 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-bin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536822 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536874 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-var-lib-kubelet\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.537179 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641871f0-a0d5-4be1-b2d0-3c7603211724-tmp-dir\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.537276 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a85ea4e-16f2-4088-ba73-7e996f46e151-host\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.537352 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-system-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.537365 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-systemd-units\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.537396 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-dbus\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.537404 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-hostroot\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.538361 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.536675 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-conf\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.538925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.537468 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-etc-kubernetes\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.538925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.537564 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.538925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.537911 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a85ea4e-16f2-4088-ba73-7e996f46e151-serviceca\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.538925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.537912 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-modprobe-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.538925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.538119 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-env-overrides\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.538925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.538528 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-kubelet-config\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:43.538925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.538595 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-socket-dir-parent\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.538925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.538621 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-config\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.538925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.538657 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-ovn\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.538925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.538894 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-system-cni-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.539441 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.538985 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-netns\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.539441 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.539040 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-kubelet\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.539441 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.539248 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:43.539441 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.539276 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.539674 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.539653 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:44.039623501 +0000 UTC m=+2.256589314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:43.539735 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.539675 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-registration-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.539909 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.539889 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:41:43.539982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.539955 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wqqrq" Oct 30 21:41:43.540752 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.540733 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-tmp\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.540838 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.540789 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-tuned\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.540838 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.540810 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d29a054-4a50-4709-81c8-01c50df2944e-ovn-node-metrics-cert\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.544812 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.544796 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:41:43.545197 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545180 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p7xlk\"" Oct 30 21:41:43.545305 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545291 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:41:43.545368 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545351 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:41:43.545419 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545363 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7tj7w\"" Oct 30 21:41:43.545419 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545350 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-msnq6\"" Oct 30 21:41:43.545419 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545403 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:41:43.545419 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545415 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:41:43.545580 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545405 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:41:43.545580 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545357 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:41:43.545580 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545435 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vtv5j\"" Oct 30 21:41:43.545580 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545399 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:41:43.545580 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545441 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:41:43.545580 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545483 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:41:43.545781 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545634 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:41:43.545781 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545640 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:41:43.545781 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545656 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:41:43.545781 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545689 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zffrj\"" Oct 30 21:41:43.545781 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545715 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:41:43.545781 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.545676 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pkpl9\"" Oct 30 21:41:43.548358 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.548335 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-cni-binary-copy\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.548645 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.548625 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40cea1c9-e71c-4451-926c-398f36e9691c-agent-certs\") pod \"konnectivity-agent-lhsfr\" (UID: \"40cea1c9-e71c-4451-926c-398f36e9691c\") " pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:43.548715 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.548629 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-daemon-config\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.554931 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.554909 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5200dd70-ca15-4547-b344-c772e014e92b-iptables-alerter-script\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.555055 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.555037 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40cea1c9-e71c-4451-926c-398f36e9691c-konnectivity-ca\") pod \"konnectivity-agent-lhsfr\" (UID: \"40cea1c9-e71c-4451-926c-398f36e9691c\") " pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:43.555149 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.555132 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-binary-copy\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.555183 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.555133 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.594879 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.594836 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:43.594879 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.594855 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:43.594879 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.594866 2581 projected.go:196] Error preparing data for projected volume kube-api-access-gfdbg for pod openshift-network-diagnostics/network-check-target-jnj6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:43.595026 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:43.594919 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg podName:a403f7b4-a616-4807-8263-678045389f84 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:44.094903023 +0000 UTC m=+2.311868841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gfdbg" (UniqueName: "kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg") pod "network-check-target-jnj6f" (UID: "a403f7b4-a616-4807-8263-678045389f84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:43.597873 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.597851 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqcc\" (UniqueName: \"kubernetes.io/projected/61f77d06-9723-4c98-8dd8-80c01d978e2f-kube-api-access-zgqcc\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.598554 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.598528 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgn5s\" (UniqueName: \"kubernetes.io/projected/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kube-api-access-lgn5s\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.601883 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.601861 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfw6m\" (UniqueName: \"kubernetes.io/projected/a749e90b-1748-48e1-b809-89cece3bab0a-kube-api-access-zfw6m\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:43.602550 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.602534 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqm4k\" (UniqueName: \"kubernetes.io/projected/641871f0-a0d5-4be1-b2d0-3c7603211724-kube-api-access-gqm4k\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.605040 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.605003 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg55q\" (UniqueName: \"kubernetes.io/projected/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-kube-api-access-dg55q\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.609884 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.609861 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6p77\" (UniqueName: \"kubernetes.io/projected/3a85ea4e-16f2-4088-ba73-7e996f46e151-kube-api-access-z6p77\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.610158 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.610136 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54q8h\" (UniqueName: \"kubernetes.io/projected/1d29a054-4a50-4709-81c8-01c50df2944e-kube-api-access-54q8h\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.610257 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.610236 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gq6m\" (UniqueName: \"kubernetes.io/projected/1a631286-ec63-4b0a-8fc6-f97280e181a1-kube-api-access-7gq6m\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.612810 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.612793 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8lr5\" (UniqueName: \"kubernetes.io/projected/5200dd70-ca15-4547-b344-c772e014e92b-kube-api-access-q8lr5\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.724958 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.724924 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:41:43.732698 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.732678 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:41:43.738112 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.738094 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:41:43.755771 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.755752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:41:43.766529 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.766509 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:41:43.772345 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.772324 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:43.781200 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.781180 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:41:43.802172 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.802153 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5xj4" Oct 30 21:41:43.806894 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.806879 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:41:43.869388 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:43.869351 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff30ba81bde53b1c4da7b24cd2c2a4b9.slice/crio-79290572886bebe8b8d943a25b6c65463c351392ee2855ef3aa21f15f9ba35ca WatchSource:0}: Error finding container 79290572886bebe8b8d943a25b6c65463c351392ee2855ef3aa21f15f9ba35ca: Status 404 returned error can't find the container with id 79290572886bebe8b8d943a25b6c65463c351392ee2855ef3aa21f15f9ba35ca Oct 30 21:41:43.869730 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:43.869710 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcbf966d484c9290a6f563fe77e5e16.slice/crio-2feb157933a68a87a88111f706be7e01313449d1e3b7f6d6f3792a8a10e740a1 WatchSource:0}: Error finding container 2feb157933a68a87a88111f706be7e01313449d1e3b7f6d6f3792a8a10e740a1: Status 404 returned error can't find the container with id 2feb157933a68a87a88111f706be7e01313449d1e3b7f6d6f3792a8a10e740a1 Oct 30 21:41:43.875183 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.875169 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:41:43.936523 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:43.936505 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:44.039254 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:44.039232 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:44.039355 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:44.039345 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:44.039405 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:44.039395 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret podName:70fbe0ee-70e4-47b1-8094-4faa96292c6b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:45.039381978 +0000 UTC m=+3.256347778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret") pod "global-pull-secret-syncer-z258k" (UID: "70fbe0ee-70e4-47b1-8094-4faa96292c6b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:44.053838 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:44.053813 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:44.140228 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:44.140171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:44.140228 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:44.140204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:44.140761 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:44.140310 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:44.140761 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:44.140322 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:44.140761 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:44.140334 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:44.140761 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:44.140344 2581 projected.go:196] Error preparing data for projected volume kube-api-access-gfdbg for pod openshift-network-diagnostics/network-check-target-jnj6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:44.140761 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:44.140362 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:45.14034956 +0000 UTC m=+3.357315365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:44.140761 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:44.140378 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg podName:a403f7b4-a616-4807-8263-678045389f84 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:45.140371449 +0000 UTC m=+3.357337249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfdbg" (UniqueName: "kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg") pod "network-check-target-jnj6f" (UID: "a403f7b4-a616-4807-8263-678045389f84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:44.269415 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:44.269370 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" event={"ID":"bfcbf966d484c9290a6f563fe77e5e16","Type":"ContainerStarted","Data":"2feb157933a68a87a88111f706be7e01313449d1e3b7f6d6f3792a8a10e740a1"} Oct 30 21:41:44.270272 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:44.270255 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" event={"ID":"ff30ba81bde53b1c4da7b24cd2c2a4b9","Type":"ContainerStarted","Data":"79290572886bebe8b8d943a25b6c65463c351392ee2855ef3aa21f15f9ba35ca"} Oct 30 21:41:44.544766 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:44.544673 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:43 +0000 UTC" deadline="2028-07-23 15:50:32.93809326 +0000 UTC" Oct 30 21:41:44.544766 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:44.544701 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15162h8m48.393395057s" Oct 30 21:41:44.596526 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:44.596490 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d29a054_4a50_4709_81c8_01c50df2944e.slice/crio-24f4015c9e24b719f689ed33f23316bcb2f8fa795e733e8f856206bd1796c34c WatchSource:0}: Error finding container 24f4015c9e24b719f689ed33f23316bcb2f8fa795e733e8f856206bd1796c34c: Status 404 returned error can't find the container with id 24f4015c9e24b719f689ed33f23316bcb2f8fa795e733e8f856206bd1796c34c Oct 30 21:41:44.659950 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:44.659928 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:44.683062 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:44.683035 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cea1c9_e71c_4451_926c_398f36e9691c.slice/crio-553b89959a6df1f69f57893dd4ef378776ca7d9d88258ef9e4dc33d9de386559 WatchSource:0}: Error finding container 553b89959a6df1f69f57893dd4ef378776ca7d9d88258ef9e4dc33d9de386559: Status 404 returned error can't find the container with id 553b89959a6df1f69f57893dd4ef378776ca7d9d88258ef9e4dc33d9de386559 Oct 30 21:41:44.704371 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:44.704327 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a85ea4e_16f2_4088_ba73_7e996f46e151.slice/crio-4402b9278f43b03f38aa205cfe2c5e907880a091a48e006854a0a73b269d0a00 WatchSource:0}: Error finding container 4402b9278f43b03f38aa205cfe2c5e907880a091a48e006854a0a73b269d0a00: Status 404 returned error can't find the container with id 4402b9278f43b03f38aa205cfe2c5e907880a091a48e006854a0a73b269d0a00 Oct 30 21:41:44.704909 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:44.704809 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5528ca3c_d24a_485c_b896_54fdc2fdb5ac.slice/crio-ca36c6a2f3eeb0ef49de9429261ceaa55020e345526187e5916069c2348176a4 WatchSource:0}: Error finding container ca36c6a2f3eeb0ef49de9429261ceaa55020e345526187e5916069c2348176a4: Status 404 returned error can't find the container with id ca36c6a2f3eeb0ef49de9429261ceaa55020e345526187e5916069c2348176a4 Oct 30 21:41:44.711406 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:44.711385 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641871f0_a0d5_4be1_b2d0_3c7603211724.slice/crio-243a34521891edabbd7609abf6d5ece7bb4ed4494f514564827a4ac45e8b0c90 WatchSource:0}: Error finding container 243a34521891edabbd7609abf6d5ece7bb4ed4494f514564827a4ac45e8b0c90: Status 404 returned error can't find the container with id 243a34521891edabbd7609abf6d5ece7bb4ed4494f514564827a4ac45e8b0c90 Oct 30 21:41:44.715874 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:44.715830 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5200dd70_ca15_4547_b344_c772e014e92b.slice/crio-2698d38b98c46a6a617df6ad41dc10fc07f6425a4a63526e494ca8b94ea24409 WatchSource:0}: Error finding container 2698d38b98c46a6a617df6ad41dc10fc07f6425a4a63526e494ca8b94ea24409: Status 404 returned error can't find the container with id 2698d38b98c46a6a617df6ad41dc10fc07f6425a4a63526e494ca8b94ea24409 Oct 30 21:41:44.718336 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:44.718233 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4b501f_ff9f_4ed7_9b82_68d5fd929cf0.slice/crio-246340e635e4734753d5659a7e8d304bbea3282b6444ff08444b7619e28e0958 WatchSource:0}: Error finding container 246340e635e4734753d5659a7e8d304bbea3282b6444ff08444b7619e28e0958: Status 404 returned error can't find the container with id 246340e635e4734753d5659a7e8d304bbea3282b6444ff08444b7619e28e0958 Oct 30 21:41:44.719124 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:44.719106 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f77d06_9723_4c98_8dd8_80c01d978e2f.slice/crio-2887fb30fa02dade9d788393214b43196c1a287e5007b66a7068d4ab278b93cd WatchSource:0}: Error finding container 2887fb30fa02dade9d788393214b43196c1a287e5007b66a7068d4ab278b93cd: Status 404 returned error can't find the container with id 2887fb30fa02dade9d788393214b43196c1a287e5007b66a7068d4ab278b93cd Oct 30 21:41:44.825474 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:41:44.825447 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a631286_ec63_4b0a_8fc6_f97280e181a1.slice/crio-79b26fad3b2a36dc9b8d8aef472ed8513da4cba7a5c37647a7f90832a85d147a WatchSource:0}: Error finding container 79b26fad3b2a36dc9b8d8aef472ed8513da4cba7a5c37647a7f90832a85d147a: Status 404 returned error can't find the container with id 79b26fad3b2a36dc9b8d8aef472ed8513da4cba7a5c37647a7f90832a85d147a Oct 30 21:41:45.048501 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.048466 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:45.048670 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.048595 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:45.048670 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.048659 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret podName:70fbe0ee-70e4-47b1-8094-4faa96292c6b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:47.048642105 +0000 UTC m=+5.265607903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret") pod "global-pull-secret-syncer-z258k" (UID: "70fbe0ee-70e4-47b1-8094-4faa96292c6b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:45.149509 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.149435 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:45.149509 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.149481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:45.150184 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.149636 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:45.150184 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.149651 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:45.150184 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.149663 2581 projected.go:196] Error preparing data for projected volume kube-api-access-gfdbg for pod openshift-network-diagnostics/network-check-target-jnj6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:45.150184 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.149714 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg podName:a403f7b4-a616-4807-8263-678045389f84 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:47.149697192 +0000 UTC m=+5.366663013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfdbg" (UniqueName: "kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg") pod "network-check-target-jnj6f" (UID: "a403f7b4-a616-4807-8263-678045389f84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:45.150184 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.149792 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:45.150184 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.149828 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:47.149816792 +0000 UTC m=+5.366782606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:45.265698 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.265664 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:45.265863 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.265813 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:41:45.265937 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.265897 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:45.266010 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.265973 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:41:45.266079 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.266057 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:45.266150 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:45.266133 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:41:45.274149 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.274118 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" event={"ID":"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0","Type":"ContainerStarted","Data":"246340e635e4734753d5659a7e8d304bbea3282b6444ff08444b7619e28e0958"} Oct 30 21:41:45.275779 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.275753 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xlpzs" event={"ID":"5200dd70-ca15-4547-b344-c772e014e92b","Type":"ContainerStarted","Data":"2698d38b98c46a6a617df6ad41dc10fc07f6425a4a63526e494ca8b94ea24409"} Oct 30 21:41:45.277286 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.277250 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5xm5p" event={"ID":"641871f0-a0d5-4be1-b2d0-3c7603211724","Type":"ContainerStarted","Data":"243a34521891edabbd7609abf6d5ece7bb4ed4494f514564827a4ac45e8b0c90"} Oct 30 21:41:45.278632 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.278574 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" event={"ID":"5528ca3c-d24a-485c-b896-54fdc2fdb5ac","Type":"ContainerStarted","Data":"ca36c6a2f3eeb0ef49de9429261ceaa55020e345526187e5916069c2348176a4"} Oct 30 21:41:45.281844 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.281820 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fgq49" event={"ID":"3a85ea4e-16f2-4088-ba73-7e996f46e151","Type":"ContainerStarted","Data":"4402b9278f43b03f38aa205cfe2c5e907880a091a48e006854a0a73b269d0a00"} Oct 30 21:41:45.283167 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.283111 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lhsfr" event={"ID":"40cea1c9-e71c-4451-926c-398f36e9691c","Type":"ContainerStarted","Data":"553b89959a6df1f69f57893dd4ef378776ca7d9d88258ef9e4dc33d9de386559"} Oct 30 21:41:45.284447 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.284422 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5xj4" event={"ID":"1a631286-ec63-4b0a-8fc6-f97280e181a1","Type":"ContainerStarted","Data":"79b26fad3b2a36dc9b8d8aef472ed8513da4cba7a5c37647a7f90832a85d147a"} Oct 30 21:41:45.285738 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.285712 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qthxc" event={"ID":"61f77d06-9723-4c98-8dd8-80c01d978e2f","Type":"ContainerStarted","Data":"2887fb30fa02dade9d788393214b43196c1a287e5007b66a7068d4ab278b93cd"} Oct 30 21:41:45.287124 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.287104 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" event={"ID":"1d29a054-4a50-4709-81c8-01c50df2944e","Type":"ContainerStarted","Data":"24f4015c9e24b719f689ed33f23316bcb2f8fa795e733e8f856206bd1796c34c"} Oct 30 21:41:45.545213 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.545100 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:43 +0000 UTC" deadline="2028-08-12 14:10:48.35605474 +0000 UTC" Oct 30 21:41:45.545213 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:45.545165 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15640h29m2.810893911s" Oct 30 21:41:47.066988 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:47.066946 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:47.067542 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.067136 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:47.067542 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.067251 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret podName:70fbe0ee-70e4-47b1-8094-4faa96292c6b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:51.067230176 +0000 UTC m=+9.284195981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret") pod "global-pull-secret-syncer-z258k" (UID: "70fbe0ee-70e4-47b1-8094-4faa96292c6b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:47.167678 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:47.167635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:47.167678 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:47.167694 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:47.168117 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.167882 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:47.168117 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.167901 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:47.168117 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.167914 2581 projected.go:196] Error preparing data for projected volume kube-api-access-gfdbg for pod openshift-network-diagnostics/network-check-target-jnj6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:47.168117 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.167969 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg podName:a403f7b4-a616-4807-8263-678045389f84 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:51.167949782 +0000 UTC m=+9.384915589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfdbg" (UniqueName: "kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg") pod "network-check-target-jnj6f" (UID: "a403f7b4-a616-4807-8263-678045389f84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:47.168117 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.168072 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:47.168117 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.168108 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:51.168095198 +0000 UTC m=+9.385061002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:47.266020 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:47.265466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:47.266020 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.265616 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:41:47.266020 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:47.265695 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:47.266020 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.265769 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:41:47.266020 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:47.265828 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:47.266020 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:47.265896 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:41:47.315901 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:47.315188 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" event={"ID":"ff30ba81bde53b1c4da7b24cd2c2a4b9","Type":"ContainerStarted","Data":"02992f0cea2db73002f0e7f8f7902ced6eb157b2226f9c542724ca34808b5f4c"} Oct 30 21:41:48.337305 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:48.337226 2581 generic.go:356] "Generic (PLEG): container finished" podID="bfcbf966d484c9290a6f563fe77e5e16" containerID="dde92ce8da66cd44cb1c4981d23d0887f7704ef8e0ae5c7288983e44bb35787f" exitCode=0 Oct 30 21:41:48.338219 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:48.338195 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" event={"ID":"bfcbf966d484c9290a6f563fe77e5e16","Type":"ContainerDied","Data":"dde92ce8da66cd44cb1c4981d23d0887f7704ef8e0ae5c7288983e44bb35787f"} Oct 30 21:41:48.463887 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:48.463828 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" podStartSLOduration=5.4638089149999995 podStartE2EDuration="5.463808915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:47.437845356 +0000 UTC m=+5.654811179" watchObservedRunningTime="2026-10-30 21:41:48.463808915 +0000 UTC m=+6.680774740" Oct 30 21:41:49.266162 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:49.266128 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:49.266419 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:49.266284 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:41:49.266419 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:49.266379 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:49.266692 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:49.266484 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:41:49.266692 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:49.266523 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:49.266692 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:49.266591 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:41:49.349340 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:49.349261 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" event={"ID":"bfcbf966d484c9290a6f563fe77e5e16","Type":"ContainerStarted","Data":"8fb4a10cf955fb34ba19be003d9983e883b4dd2a4e93a71e81e42ecb875d0801"} Oct 30 21:41:49.356119 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:49.356089 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="f63dbf97bf7747c792f8f516e2c326d0d58f263bb77113d6102756b406c08527" exitCode=0 Oct 30 21:41:49.356227 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:49.356148 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" event={"ID":"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0","Type":"ContainerDied","Data":"f63dbf97bf7747c792f8f516e2c326d0d58f263bb77113d6102756b406c08527"} Oct 30 21:41:49.602800 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:49.602738 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" podStartSLOduration=6.602719698 podStartE2EDuration="6.602719698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:49.47534857 +0000 UTC m=+7.692314395" watchObservedRunningTime="2026-10-30 21:41:49.602719698 +0000 UTC m=+7.819685522" Oct 30 21:41:50.365314 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:50.365277 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lhsfr" event={"ID":"40cea1c9-e71c-4451-926c-398f36e9691c","Type":"ContainerStarted","Data":"5b573e0acf15633b4c36d443c4e52dae7c006c9c7ab43871470a152f8f6dbb26"} Oct 30 21:41:51.098766 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.098690 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:51.098920 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.098865 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:51.098995 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.098929 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret podName:70fbe0ee-70e4-47b1-8094-4faa96292c6b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:59.098911393 +0000 UTC m=+17.315877197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret") pod "global-pull-secret-syncer-z258k" (UID: "70fbe0ee-70e4-47b1-8094-4faa96292c6b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:51.199768 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.199733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:51.199885 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.199852 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:51.199993 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.199976 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:51.200044 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.200039 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:59.200021971 +0000 UTC m=+17.416987772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:51.200145 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.200131 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:51.200189 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.200155 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:51.200189 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.200167 2581 projected.go:196] Error preparing data for projected volume kube-api-access-gfdbg for pod openshift-network-diagnostics/network-check-target-jnj6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:51.200284 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.200207 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg podName:a403f7b4-a616-4807-8263-678045389f84 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:59.200195119 +0000 UTC m=+17.417160932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfdbg" (UniqueName: "kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg") pod "network-check-target-jnj6f" (UID: "a403f7b4-a616-4807-8263-678045389f84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:51.265801 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.265766 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:51.265962 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.265781 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:51.265962 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.265917 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:41:51.265962 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.265959 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:51.266137 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.266059 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:41:51.266178 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:51.266147 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:41:51.370198 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.370172 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" event={"ID":"5528ca3c-d24a-485c-b896-54fdc2fdb5ac","Type":"ContainerStarted","Data":"e10cdde71638b49d9f38b0bcef2db6b6a46a5822ffe4933c969284f861bb2db5"} Oct 30 21:41:51.372199 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.372174 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fgq49" event={"ID":"3a85ea4e-16f2-4088-ba73-7e996f46e151","Type":"ContainerStarted","Data":"ac6f5da83f306b11aaec4d9748ad7b309d464237c208f8be7831c54ed45a90b7"} Oct 30 21:41:51.483389 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.483325 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fgq49" podStartSLOduration=2.539541637 podStartE2EDuration="8.483305485s" totalImagesPullingTime="5.943763848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:51.482783137 +0000 UTC m=+9.699748958" watchObservedRunningTime="2026-10-30 21:41:51.483305485 +0000 UTC m=+9.700271306" Oct 30 21:41:51.483742 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.483709 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lhsfr" podStartSLOduration=3.819163653 podStartE2EDuration="8.483699967s" totalImagesPullingTime="4.664536314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:50.489539901 +0000 UTC m=+8.706505725" watchObservedRunningTime="2026-10-30 21:41:51.483699967 +0000 UTC m=+9.700665787" Oct 30 21:41:51.883472 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.883440 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:51.884102 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:51.884078 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:53.265912 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:53.265881 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:53.266376 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:53.266015 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:41:53.266376 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:53.266091 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:53.266376 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:53.266166 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:41:53.266376 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:53.266228 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:53.266376 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:53.266291 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:41:53.345213 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:53.345100 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:41:53.381572 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:53.381539 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" event={"ID":"5528ca3c-d24a-485c-b896-54fdc2fdb5ac","Type":"ContainerStarted","Data":"d9b7cf8fbf2e077efb8e6e1dfb9bd0635231850a7f027f0c1d55ee888feb61f8"} Oct 30 21:41:53.385288 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:53.385215 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:41:53.386217 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:53.386193 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5xj4" event={"ID":"1a631286-ec63-4b0a-8fc6-f97280e181a1","Type":"ContainerStarted","Data":"103fc1dd4a2b14ef9ae2b62c1e6f13965268f6fc9df2fa8aa001a439243e062a"} Oct 30 21:41:53.504607 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:53.504548 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-n5xj4" podStartSLOduration=2.312249612 podStartE2EDuration="10.504531528s" totalImagesPullingTime="8.192281916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:53.503776916 +0000 UTC m=+11.720742739" watchObservedRunningTime="2026-10-30 21:41:53.504531528 +0000 UTC m=+11.721497353" Oct 30 21:41:54.185048 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:54.184715 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:41:53.345119795Z","UUID":"5de1dbe6-a10f-4375-9d80-dfcb812cfa0b","Handler":null,"Name":"","Endpoint":""} Oct 30 21:41:54.186791 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:54.186762 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:41:54.186791 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:54.186795 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:41:54.391267 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:54.391054 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5xm5p" event={"ID":"641871f0-a0d5-4be1-b2d0-3c7603211724","Type":"ContainerStarted","Data":"abefaa94e199bee40b4921e54c3560c28c57dde521d857a45b2d7e39571a8835"} Oct 30 21:41:54.395168 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:54.395102 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" event={"ID":"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0","Type":"ContainerStarted","Data":"d3e76d9b314584050fc902be93d7a55365595269ff9e7f3a6f062a79d8bc8270"} Oct 30 21:41:54.627858 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:54.627769 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5xm5p" podStartSLOduration=2.808536468 podStartE2EDuration="11.627755715s" totalImagesPullingTime="8.819219247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:54.507919821 +0000 UTC m=+12.724885641" watchObservedRunningTime="2026-10-30 21:41:54.627755715 +0000 UTC m=+12.844721534" Oct 30 21:41:55.265539 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:55.265509 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:55.265698 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:55.265591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:55.265744 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:55.265723 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:41:55.265776 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:55.265759 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:55.265829 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:55.265809 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:41:55.265931 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:55.265855 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:41:55.399837 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:55.399652 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="d3e76d9b314584050fc902be93d7a55365595269ff9e7f3a6f062a79d8bc8270" exitCode=0 Oct 30 21:41:55.399837 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:55.399742 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" event={"ID":"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0","Type":"ContainerDied","Data":"d3e76d9b314584050fc902be93d7a55365595269ff9e7f3a6f062a79d8bc8270"} Oct 30 21:41:55.402144 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:55.402117 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xlpzs" event={"ID":"5200dd70-ca15-4547-b344-c772e014e92b","Type":"ContainerStarted","Data":"39317b5aa9eba25723cf7aa86c286683bf9c0040a7d00fae9e6b53122861440d"} Oct 30 21:41:55.404797 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:55.404763 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" event={"ID":"5528ca3c-d24a-485c-b896-54fdc2fdb5ac","Type":"ContainerStarted","Data":"6ea385c62c0bdcdfc18a27e64e783207e397a18d445cb1c81587c285403485e4"} Oct 30 21:41:55.688882 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:55.688778 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xlpzs" podStartSLOduration=3.590477808 podStartE2EDuration="12.688759896s" totalImagesPullingTime="9.098282088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:55.688197369 +0000 UTC m=+13.905163191" watchObservedRunningTime="2026-10-30 21:41:55.688759896 +0000 UTC m=+13.905725716" Oct 30 21:41:55.806465 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:55.806411 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" podStartSLOduration=3.075943242 podStartE2EDuration="12.806394684s" totalImagesPullingTime="9.730451442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:55.806382514 +0000 UTC m=+14.023348334" watchObservedRunningTime="2026-10-30 21:41:55.806394684 +0000 UTC m=+14.023360525" Oct 30 21:41:56.408929 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:56.408891 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qthxc" event={"ID":"61f77d06-9723-4c98-8dd8-80c01d978e2f","Type":"ContainerStarted","Data":"03313eebe3871ce877d21dd177ad460621c030141a5d1f88d367d04db3fb25c7"} Oct 30 21:41:57.265418 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:57.265386 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:57.265418 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:57.265412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:57.265418 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:57.265403 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:57.265681 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:57.265507 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:41:57.265681 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:57.265594 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:41:57.265681 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:57.265643 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:41:57.364031 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:57.363997 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:57.364180 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:57.364130 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:41:57.364753 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:57.364731 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:41:57.413447 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:57.413414 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="8c1c8b32b85fa50bbbd21004b1aecceddbefb286686403b812a86a98a3cebc8e" exitCode=0 Oct 30 21:41:57.413876 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:57.413504 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" event={"ID":"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0","Type":"ContainerDied","Data":"8c1c8b32b85fa50bbbd21004b1aecceddbefb286686403b812a86a98a3cebc8e"} Oct 30 21:41:57.482056 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:57.482000 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qthxc" podStartSLOduration=3.800815438 podStartE2EDuration="14.481983199s" totalImagesPullingTime="10.681167761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:56.526049163 +0000 UTC m=+14.743014986" watchObservedRunningTime="2026-10-30 21:41:57.481983199 +0000 UTC m=+15.698949016" Oct 30 21:41:59.161612 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:59.161574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:59.162031 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.161701 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:59.162031 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.161776 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret podName:70fbe0ee-70e4-47b1-8094-4faa96292c6b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.161755487 +0000 UTC m=+33.378721291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret") pod "global-pull-secret-syncer-z258k" (UID: "70fbe0ee-70e4-47b1-8094-4faa96292c6b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:59.262486 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:59.262449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:59.262671 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:59.262504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:59.262733 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.262683 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:59.262733 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.262700 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:59.262733 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.262713 2581 projected.go:196] Error preparing data for projected volume kube-api-access-gfdbg for pod openshift-network-diagnostics/network-check-target-jnj6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:59.262848 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.262776 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg podName:a403f7b4-a616-4807-8263-678045389f84 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.262757503 +0000 UTC m=+33.479723305 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfdbg" (UniqueName: "kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg") pod "network-check-target-jnj6f" (UID: "a403f7b4-a616-4807-8263-678045389f84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:59.263324 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.263302 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:59.263422 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.263365 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.263349695 +0000 UTC m=+33.480315497 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:59.266160 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:59.266137 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:41:59.266330 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.266301 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:41:59.266444 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:59.266388 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:41:59.266527 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.266504 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:41:59.266611 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:59.266592 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:41:59.266697 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:41:59.266677 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:41:59.421198 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:59.421114 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="271a66aba1ffdf066da8f5365b53d8bb4f8e0c96240e05e6671d0a18972ba145" exitCode=0 Oct 30 21:41:59.421198 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:41:59.421176 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" event={"ID":"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0","Type":"ContainerDied","Data":"271a66aba1ffdf066da8f5365b53d8bb4f8e0c96240e05e6671d0a18972ba145"} Oct 30 21:42:00.426448 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:00.426271 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" event={"ID":"1d29a054-4a50-4709-81c8-01c50df2944e","Type":"ContainerStarted","Data":"94ac79caf824ab8fe0c444ab66cbb019c815d75c481647ef2fc41a71a7353d0a"} Oct 30 21:42:00.426877 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:00.426481 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" event={"ID":"1d29a054-4a50-4709-81c8-01c50df2944e","Type":"ContainerStarted","Data":"35d7266a333136d7027506aa5a65a783bcbb67748312e366bdb659c39fe90066"} Oct 30 21:42:00.426877 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:00.426512 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" event={"ID":"1d29a054-4a50-4709-81c8-01c50df2944e","Type":"ContainerStarted","Data":"7252e12eaa1761becf499f155f403ab2fad9cfcfb960335be829bb00367ddecb"} Oct 30 21:42:00.426877 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:00.426540 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" event={"ID":"1d29a054-4a50-4709-81c8-01c50df2944e","Type":"ContainerStarted","Data":"5f7489844cfba1c03380a54bb47a290315166ef2824d8639bedcb5b9a64003f1"} Oct 30 21:42:00.426877 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:00.426568 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" event={"ID":"1d29a054-4a50-4709-81c8-01c50df2944e","Type":"ContainerStarted","Data":"4ef1283a1fb7d99bcf99c765608f5d8f3e3b15ba61f11a0e9bbd16b6149f2276"} Oct 30 21:42:00.426877 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:00.426596 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" event={"ID":"1d29a054-4a50-4709-81c8-01c50df2944e","Type":"ContainerStarted","Data":"af77929955ef5bbccc3cc2a7359ab60ae778e3308fbf8e0ac0e716b7db399a94"} Oct 30 21:42:01.265759 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:01.265723 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:01.265915 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:01.265766 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:01.265915 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:01.265732 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:01.265915 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:01.265867 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:42:01.266063 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:01.265929 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:42:01.266063 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:01.266025 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:42:02.433442 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:02.433405 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" event={"ID":"1d29a054-4a50-4709-81c8-01c50df2944e","Type":"ContainerStarted","Data":"68bdc89f5dda11a1cfb92b28740191363f31d71cf81b8efa51bff71ce7eb6c41"} Oct 30 21:42:02.435911 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:02.435886 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="c5a8a1aeea37c25b03509f2da19ab118499767d54436e29b46f96e68ded1b815" exitCode=0 Oct 30 21:42:02.436032 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:02.435927 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" event={"ID":"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0","Type":"ContainerDied","Data":"c5a8a1aeea37c25b03509f2da19ab118499767d54436e29b46f96e68ded1b815"} Oct 30 21:42:03.265633 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:03.265605 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:03.265633 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:03.265639 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:03.265830 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:03.265641 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:03.265830 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:03.265706 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:42:03.265830 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:03.265771 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:42:03.265920 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:03.265841 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:42:03.440613 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:03.440581 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="d8e933ad56e4eb3e7ccbf76221ad2eebfdf37847e0e15f82fa13fa6a2f513e28" exitCode=0 Oct 30 21:42:03.440998 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:03.440653 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" event={"ID":"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0","Type":"ContainerDied","Data":"d8e933ad56e4eb3e7ccbf76221ad2eebfdf37847e0e15f82fa13fa6a2f513e28"} Oct 30 21:42:04.446603 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:04.446498 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" event={"ID":"1d29a054-4a50-4709-81c8-01c50df2944e","Type":"ContainerStarted","Data":"bc0fb8817aff8c372d5857394dd5fbb3374f3eef0faf44b368b5e84b80ca2c21"} Oct 30 21:42:04.447207 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:04.446859 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:04.447207 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:04.446932 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:04.447207 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:04.447002 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:04.451110 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:04.451071 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" event={"ID":"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0","Type":"ContainerStarted","Data":"48dda79461fb840cf6ff146ead54ef3886d08ed61b6314e08dc110794f65a5f7"} Oct 30 21:42:04.463774 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:04.463747 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:04.464548 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:04.464528 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:04.669639 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:04.669588 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" podStartSLOduration=6.563478837 podStartE2EDuration="21.669576274s" totalImagesPullingTime="15.106097437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:04.619709016 +0000 UTC m=+22.836674836" watchObservedRunningTime="2026-10-30 21:42:04.669576274 +0000 UTC m=+22.886542127" Oct 30 21:42:04.743495 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:04.743410 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5g5tc" podStartSLOduration=8.557730689 podStartE2EDuration="21.743397957s" totalImagesPullingTime="13.185667268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:04.742962428 +0000 UTC m=+22.959928249" watchObservedRunningTime="2026-10-30 21:42:04.743397957 +0000 UTC m=+22.960363780" Oct 30 21:42:05.265866 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:05.265835 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:05.265866 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:05.265851 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:05.266121 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:05.265871 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:05.266121 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:05.265939 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:42:05.266121 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:05.266033 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:42:05.266245 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:05.266117 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:42:07.027752 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:07.027718 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z258k"] Oct 30 21:42:07.028230 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:07.027847 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:07.028230 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:07.027966 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:42:07.030174 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:07.030149 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jnj6f"] Oct 30 21:42:07.030357 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:07.030340 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:07.030458 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:07.030439 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:42:07.077061 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:07.077034 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vbnk5"] Oct 30 21:42:07.077200 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:07.077135 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:07.077235 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:07.077209 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:42:08.266072 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:08.266046 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:08.266386 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:08.266187 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:42:09.265885 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:09.265850 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:09.266062 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:09.265976 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:42:09.266062 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:09.266034 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:09.266496 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:09.266120 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:42:10.265897 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:10.265864 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:10.266099 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:10.266018 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z258k" podUID="70fbe0ee-70e4-47b1-8094-4faa96292c6b" Oct 30 21:42:11.265607 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:11.265574 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:11.265607 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:11.265603 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:11.266192 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:11.265709 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbnk5" podUID="a749e90b-1748-48e1-b809-89cece3bab0a" Oct 30 21:42:11.266192 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:11.265825 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnj6f" podUID="a403f7b4-a616-4807-8263-678045389f84" Oct 30 21:42:12.050198 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.050033 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeReady" Oct 30 21:42:12.050352 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.050303 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:42:12.163431 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.163398 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qcvmd"] Oct 30 21:42:12.165313 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.165291 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dq5vk"] Oct 30 21:42:12.165430 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.165412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qcvmd" Oct 30 21:42:12.166860 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.166844 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f47zh"] Oct 30 21:42:12.166986 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.166970 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dq5vk" Oct 30 21:42:12.168307 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.168288 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27"] Oct 30 21:42:12.168410 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.168394 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f47zh" Oct 30 21:42:12.169922 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.169906 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27" Oct 30 21:42:12.204408 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.204375 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw"] Oct 30 21:42:12.205899 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.205884 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.213218 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.213194 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6q9s5"] Oct 30 21:42:12.214782 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.214763 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wbpqf"] Oct 30 21:42:12.214890 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.214874 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:12.216198 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.216179 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68687f4c6c-hd5kb"] Oct 30 21:42:12.216294 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.216283 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.217527 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.217510 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76cdd49c4b-bf2dt"] Oct 30 21:42:12.217635 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.217620 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.218830 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.218814 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j"] Oct 30 21:42:12.218912 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.218897 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.220187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.220170 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr"] Oct 30 21:42:12.221090 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.220551 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:12.225101 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.225083 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-bthdr"] Oct 30 21:42:12.225215 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.225197 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:12.227020 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.226972 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.252833 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.252812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxd6\" (UniqueName: \"kubernetes.io/projected/78532fdc-d35e-4ec3-a48a-18fe0bbe8484-kube-api-access-swxd6\") pod \"kas-connection-checker-685895c7c9-qcvmd\" (UID: \"78532fdc-d35e-4ec3-a48a-18fe0bbe8484\") " pod="kube-system/kas-connection-checker-685895c7c9-qcvmd" Oct 30 21:42:12.252965 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.252863 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3bcc5627-b74e-49ed-a82b-36c2920b9917-serving-cert\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.253142 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.253112 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk7q5\" (UniqueName: \"kubernetes.io/projected/22242d39-b0fc-48a0-a9f2-9a28f7106fbd-kube-api-access-tk7q5\") pod \"volume-data-source-validator-58c9f5f46d-n6b27\" (UID: \"22242d39-b0fc-48a0-a9f2-9a28f7106fbd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27" Oct 30 21:42:12.253275 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.253256 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhwd\" (UniqueName: \"kubernetes.io/projected/aff7cebc-f18e-4b88-bb91-bec7ed9ab097-kube-api-access-wqhwd\") pod \"kas-connection-checker-685895c7c9-f47zh\" (UID: \"aff7cebc-f18e-4b88-bb91-bec7ed9ab097\") " pod="kube-system/kas-connection-checker-685895c7c9-f47zh" Oct 30 21:42:12.253386 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.253366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8gcb\" (UniqueName: \"kubernetes.io/projected/5197a548-5e64-49a3-a5b4-421898abe7df-kube-api-access-m8gcb\") pod \"kas-connection-checker-685895c7c9-dq5vk\" (UID: \"5197a548-5e64-49a3-a5b4-421898abe7df\") " pod="kube-system/kas-connection-checker-685895c7c9-dq5vk" Oct 30 21:42:12.253550 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.253497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3bcc5627-b74e-49ed-a82b-36c2920b9917-config\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.253550 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.253531 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654cq\" (UniqueName: \"kubernetes.io/projected/3bcc5627-b74e-49ed-a82b-36c2920b9917-kube-api-access-654cq\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.255094 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.255003 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks"] Oct 30 21:42:12.257928 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.257909 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc"] Oct 30 21:42:12.258097 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.258055 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.259737 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.259716 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-755f8d7949-984mn"] Oct 30 21:42:12.259855 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.259836 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc" Oct 30 21:42:12.261535 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.261515 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.261924 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.261895 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.262312 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.262293 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.262457 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.262440 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cgwj9\"" Oct 30 21:42:12.262876 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.262856 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l8jvk"] Oct 30 21:42:12.264526 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.264509 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-trg4h"] Oct 30 21:42:12.264630 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.264611 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.266331 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.266308 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:12.266893 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.266877 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:12.270386 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.270367 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qcvmd"] Oct 30 21:42:12.270471 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.270412 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f47zh"] Oct 30 21:42:12.270471 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.270434 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw"] Oct 30 21:42:12.270471 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.270456 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27"] Oct 30 21:42:12.270566 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.270480 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dq5vk"] Oct 30 21:42:12.270566 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.270500 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6q9s5"] Oct 30 21:42:12.270566 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.270519 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr"] Oct 30 21:42:12.270566 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.270545 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j"] Oct 30 21:42:12.270566 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.270564 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks"] Oct 30 21:42:12.271142 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.271001 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l8jvk"] Oct 30 21:42:12.271746 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.271730 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-755f8d7949-984mn"] Oct 30 21:42:12.303862 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.303836 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68687f4c6c-hd5kb"] Oct 30 21:42:12.304725 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.304706 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76cdd49c4b-bf2dt"] Oct 30 21:42:12.305611 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.305592 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wbpqf"] Oct 30 21:42:12.306455 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.306397 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc"] Oct 30 21:42:12.307064 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.307046 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-bthdr"] Oct 30 21:42:12.312881 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.312831 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-trg4h"] Oct 30 21:42:12.318779 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.318752 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:42:12.318890 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.318779 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.318890 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.318806 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.318890 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.318846 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:42:12.318890 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.318857 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.319168 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.318889 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.319168 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.318893 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:42:12.319168 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319071 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:42:12.319168 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319114 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.319168 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319152 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.319445 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319179 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:42:12.319445 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319247 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.319445 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319261 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.319590 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319542 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.319590 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319554 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.319678 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319608 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.319678 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319618 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.319748 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319675 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.319748 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319692 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.319807 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319780 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.319857 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319840 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.319891 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319841 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.319891 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319883 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:42:12.319972 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319889 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:42:12.319972 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.319905 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.320047 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.320023 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.320047 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.320031 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:42:12.325869 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.325854 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.354800 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.354775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7l4k\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-kube-api-access-t7l4k\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.354800 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.354802 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-tmp\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.354939 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.354819 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6307df15-18fe-45cb-b509-298bcb1fcf3b-cert\") pod \"ingress-canary-trg4h\" (UID: \"6307df15-18fe-45cb-b509-298bcb1fcf3b\") " pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:12.354939 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.354837 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pnln\" (UniqueName: \"kubernetes.io/projected/56e7aa71-e129-4ceb-91a9-06770acc40f7-kube-api-access-8pnln\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.354939 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.354860 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk7q5\" (UniqueName: \"kubernetes.io/projected/22242d39-b0fc-48a0-a9f2-9a28f7106fbd-kube-api-access-tk7q5\") pod \"volume-data-source-validator-58c9f5f46d-n6b27\" (UID: \"22242d39-b0fc-48a0-a9f2-9a28f7106fbd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27" Oct 30 21:42:12.355258 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.354962 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhwd\" (UniqueName: \"kubernetes.io/projected/aff7cebc-f18e-4b88-bb91-bec7ed9ab097-kube-api-access-wqhwd\") pod \"kas-connection-checker-685895c7c9-f47zh\" (UID: \"aff7cebc-f18e-4b88-bb91-bec7ed9ab097\") " pod="kube-system/kas-connection-checker-685895c7c9-f47zh" Oct 30 21:42:12.355258 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355013 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-snapshots\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.355258 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355049 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/81fb1745-4867-438d-b4e0-d44e743e5230-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.355258 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvjt\" (UniqueName: \"kubernetes.io/projected/81fb1745-4867-438d-b4e0-d44e743e5230-kube-api-access-4rvjt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.355258 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355140 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12730f3a-4d15-47f1-af22-9f4e37eedac7-tmp-dir\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.355258 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:12.355432 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355269 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-image-registry-private-configuration\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.355432 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-trusted-ca\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.355432 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355314 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81fb1745-4867-438d-b4e0-d44e743e5230-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.355432 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-serving-cert\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.355432 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355369 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.355432 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355414 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-trusted-ca\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.355611 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355482 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3bcc5627-b74e-49ed-a82b-36c2920b9917-config\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.355611 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355514 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-stats-auth\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.355611 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355539 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3bcc5627-b74e-49ed-a82b-36c2920b9917-serving-cert\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.355611 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.355611 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355589 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-certificates\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.355611 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355605 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mt5k\" (UniqueName: \"kubernetes.io/projected/3f03966a-97af-46f0-99a5-d50a4f54e744-kube-api-access-8mt5k\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:12.355790 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355635 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-bound-sa-token\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.355790 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355701 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/699c93ce-4424-4460-8cb2-b5db6ca24033-ca-trust-extracted\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.355849 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355787 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-image-registry-private-configuration\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.355849 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355815 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bnrn\" (UniqueName: \"kubernetes.io/projected/5387b573-d7bf-47bc-9d20-755835781b5b-kube-api-access-7bnrn\") pod \"network-check-source-6f78d59f99-l2prc\" (UID: \"5387b573-d7bf-47bc-9d20-755835781b5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc" Oct 30 21:42:12.355849 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355846 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-default-certificate\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.355935 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355894 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:12.355935 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txqg8\" (UniqueName: \"kubernetes.io/projected/65c914e8-45f5-4219-9790-b5fc849d39f7-kube-api-access-txqg8\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:12.355935 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355931 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.356044 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.355965 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.356044 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356000 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3bcc5627-b74e-49ed-a82b-36c2920b9917-config\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.356044 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8gcb\" (UniqueName: \"kubernetes.io/projected/5197a548-5e64-49a3-a5b4-421898abe7df-kube-api-access-m8gcb\") pod \"kas-connection-checker-685895c7c9-dq5vk\" (UID: \"5197a548-5e64-49a3-a5b4-421898abe7df\") " pod="kube-system/kas-connection-checker-685895c7c9-dq5vk" Oct 30 21:42:12.356128 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356055 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:12.356128 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356080 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbcw8\" (UniqueName: \"kubernetes.io/projected/6307df15-18fe-45cb-b509-298bcb1fcf3b-kube-api-access-tbcw8\") pod \"ingress-canary-trg4h\" (UID: \"6307df15-18fe-45cb-b509-298bcb1fcf3b\") " pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:12.356188 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356131 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxd6\" (UniqueName: \"kubernetes.io/projected/78532fdc-d35e-4ec3-a48a-18fe0bbe8484-kube-api-access-swxd6\") pod \"kas-connection-checker-685895c7c9-qcvmd\" (UID: \"78532fdc-d35e-4ec3-a48a-18fe0bbe8484\") " pod="kube-system/kas-connection-checker-685895c7c9-qcvmd" Oct 30 21:42:12.356188 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-installation-pull-secrets\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.356188 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12730f3a-4d15-47f1-af22-9f4e37eedac7-config-volume\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.356266 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356240 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654cq\" (UniqueName: \"kubernetes.io/projected/3bcc5627-b74e-49ed-a82b-36c2920b9917-kube-api-access-654cq\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.356266 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356260 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58245\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-kube-api-access-58245\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.356324 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b21892bd-c791-419a-a1d7-4bdc643b7f07-serving-cert\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.356353 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356324 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:12.356353 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356346 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-config\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.356409 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356363 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4f9s\" (UniqueName: \"kubernetes.io/projected/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-kube-api-access-j4f9s\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.356409 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356393 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9cgw\" (UniqueName: \"kubernetes.io/projected/12730f3a-4d15-47f1-af22-9f4e37eedac7-kube-api-access-c9cgw\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.356469 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356423 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-bound-sa-token\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.356469 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzqbm\" (UniqueName: \"kubernetes.io/projected/b21892bd-c791-419a-a1d7-4bdc643b7f07-kube-api-access-kzqbm\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.356531 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356485 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-trusted-ca\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.356531 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356508 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:12.356591 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356534 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a474868-17e7-4ad9-a3f7-65376fc52940-ca-trust-extracted\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.356591 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356571 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.356649 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356588 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3f03966a-97af-46f0-99a5-d50a4f54e744-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:12.356649 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356630 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-certificates\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.356710 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356650 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-service-ca-bundle\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.356710 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.356710 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.356707 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-installation-pull-secrets\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.369022 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.368999 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:42:12.402244 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402191 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:42:12.402244 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402280 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:42:12.402244 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402310 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:42:12.402244 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402323 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:42:12.402244 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402330 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:42:12.402244 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402349 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402409 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402419 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9wg9h\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402438 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402495 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402566 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4t2vs\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402631 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402662 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nrkss\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402682 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402683 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402709 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402718 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7fm9b\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402748 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402775 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-btj7g\"" Oct 30 21:42:12.402850 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.402754 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xlldj\"" Oct 30 21:42:12.404422 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.404400 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:42:12.404505 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.404458 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zzddg\"" Oct 30 21:42:12.404505 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.404490 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j7s9f\"" Oct 30 21:42:12.410216 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.410192 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3bcc5627-b74e-49ed-a82b-36c2920b9917-serving-cert\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.410312 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.410195 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:42:12.410312 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.410237 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:42:12.410312 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.410256 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:42:12.410489 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.410318 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7xfbb\"" Oct 30 21:42:12.410489 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.410389 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-t4n5m\"" Oct 30 21:42:12.410489 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.410477 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:42:12.410680 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.410526 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4hz4d\"" Oct 30 21:42:12.410680 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.410565 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-r22zh\"" Oct 30 21:42:12.410680 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.410653 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2skfb\"" Oct 30 21:42:12.420057 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.420026 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxd6\" (UniqueName: \"kubernetes.io/projected/78532fdc-d35e-4ec3-a48a-18fe0bbe8484-kube-api-access-swxd6\") pod \"kas-connection-checker-685895c7c9-qcvmd\" (UID: \"78532fdc-d35e-4ec3-a48a-18fe0bbe8484\") " pod="kube-system/kas-connection-checker-685895c7c9-qcvmd" Oct 30 21:42:12.420187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.420084 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654cq\" (UniqueName: \"kubernetes.io/projected/3bcc5627-b74e-49ed-a82b-36c2920b9917-kube-api-access-654cq\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.420328 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.420310 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk7q5\" (UniqueName: \"kubernetes.io/projected/22242d39-b0fc-48a0-a9f2-9a28f7106fbd-kube-api-access-tk7q5\") pod \"volume-data-source-validator-58c9f5f46d-n6b27\" (UID: \"22242d39-b0fc-48a0-a9f2-9a28f7106fbd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27" Oct 30 21:42:12.420389 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.420355 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhwd\" (UniqueName: \"kubernetes.io/projected/aff7cebc-f18e-4b88-bb91-bec7ed9ab097-kube-api-access-wqhwd\") pod \"kas-connection-checker-685895c7c9-f47zh\" (UID: \"aff7cebc-f18e-4b88-bb91-bec7ed9ab097\") " pod="kube-system/kas-connection-checker-685895c7c9-f47zh" Oct 30 21:42:12.420445 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.420422 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8gcb\" (UniqueName: \"kubernetes.io/projected/5197a548-5e64-49a3-a5b4-421898abe7df-kube-api-access-m8gcb\") pod \"kas-connection-checker-685895c7c9-dq5vk\" (UID: \"5197a548-5e64-49a3-a5b4-421898abe7df\") " pod="kube-system/kas-connection-checker-685895c7c9-dq5vk" Oct 30 21:42:12.458068 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-stats-auth\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.458068 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.458292 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-certificates\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.458292 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458106 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mt5k\" (UniqueName: \"kubernetes.io/projected/3f03966a-97af-46f0-99a5-d50a4f54e744-kube-api-access-8mt5k\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:12.458292 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458121 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-bound-sa-token\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.458292 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.458201 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:12.458292 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.458223 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd49c4b-bf2dt: secret "image-registry-tls" not found Oct 30 21:42:12.458292 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/699c93ce-4424-4460-8cb2-b5db6ca24033-ca-trust-extracted\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.458292 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-image-registry-private-configuration\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.458311 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls podName:8a474868-17e7-4ad9-a3f7-65376fc52940 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:12.958289953 +0000 UTC m=+31.175255753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls") pod "image-registry-76cdd49c4b-bf2dt" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940") : secret "image-registry-tls" not found Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnrn\" (UniqueName: \"kubernetes.io/projected/5387b573-d7bf-47bc-9d20-755835781b5b-kube-api-access-7bnrn\") pod \"network-check-source-6f78d59f99-l2prc\" (UID: \"5387b573-d7bf-47bc-9d20-755835781b5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc" Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458359 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-default-certificate\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458391 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458417 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txqg8\" (UniqueName: \"kubernetes.io/projected/65c914e8-45f5-4219-9790-b5fc849d39f7-kube-api-access-txqg8\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458530 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbcw8\" (UniqueName: \"kubernetes.io/projected/6307df15-18fe-45cb-b509-298bcb1fcf3b-kube-api-access-tbcw8\") pod \"ingress-canary-trg4h\" (UID: \"6307df15-18fe-45cb-b509-298bcb1fcf3b\") " pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458590 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-installation-pull-secrets\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.458614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458617 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12730f3a-4d15-47f1-af22-9f4e37eedac7-config-volume\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458657 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58245\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-kube-api-access-58245\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b21892bd-c791-419a-a1d7-4bdc643b7f07-serving-cert\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458687 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/699c93ce-4424-4460-8cb2-b5db6ca24033-ca-trust-extracted\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-config\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458767 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4f9s\" (UniqueName: \"kubernetes.io/projected/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-kube-api-access-j4f9s\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.458782 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458794 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cgw\" (UniqueName: \"kubernetes.io/projected/12730f3a-4d15-47f1-af22-9f4e37eedac7-kube-api-access-c9cgw\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458801 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-certificates\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.458830 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls podName:3f03966a-97af-46f0-99a5-d50a4f54e744 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:12.958816475 +0000 UTC m=+31.175782274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xmt2j" (UID: "3f03966a-97af-46f0-99a5-d50a4f54e744") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-bound-sa-token\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzqbm\" (UniqueName: \"kubernetes.io/projected/b21892bd-c791-419a-a1d7-4bdc643b7f07-kube-api-access-kzqbm\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458901 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-trusted-ca\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458935 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:12.459187 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.458978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a474868-17e7-4ad9-a3f7-65376fc52940-ca-trust-extracted\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459010 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3f03966a-97af-46f0-99a5-d50a4f54e744-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459069 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-certificates\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459098 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-service-ca-bundle\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459165 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-installation-pull-secrets\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7l4k\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-kube-api-access-t7l4k\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459197 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-tmp\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459248 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6307df15-18fe-45cb-b509-298bcb1fcf3b-cert\") pod \"ingress-canary-trg4h\" (UID: \"6307df15-18fe-45cb-b509-298bcb1fcf3b\") " pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459280 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pnln\" (UniqueName: \"kubernetes.io/projected/56e7aa71-e129-4ceb-91a9-06770acc40f7-kube-api-access-8pnln\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-snapshots\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/81fb1745-4867-438d-b4e0-d44e743e5230-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.459352 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:12.459879 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvjt\" (UniqueName: \"kubernetes.io/projected/81fb1745-4867-438d-b4e0-d44e743e5230-kube-api-access-4rvjt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.459400 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls podName:65c914e8-45f5-4219-9790-b5fc849d39f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:12.959386163 +0000 UTC m=+31.176351961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jhjqr" (UID: "65c914e8-45f5-4219-9790-b5fc849d39f7") : secret "samples-operator-tls" not found Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12730f3a-4d15-47f1-af22-9f4e37eedac7-tmp-dir\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.459498 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-image-registry-private-configuration\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459537 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-trusted-ca\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.459549 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:12.959532421 +0000 UTC m=+31.176498220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : secret "router-metrics-certs-default" not found Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81fb1745-4867-438d-b4e0-d44e743e5230-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-serving-cert\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-trusted-ca\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.460475 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459732 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12730f3a-4d15-47f1-af22-9f4e37eedac7-tmp-dir\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.461038 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.460764 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12730f3a-4d15-47f1-af22-9f4e37eedac7-config-volume\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.461536 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.461511 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-trusted-ca\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.461639 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.461618 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-stats-auth\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.461936 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.461914 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-image-registry-private-configuration\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.462066 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.462024 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-tmp\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.462199 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.462137 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-certificates\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.462407 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.462380 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:12.462516 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.462408 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68687f4c6c-hd5kb: secret "image-registry-tls" not found Oct 30 21:42:12.462569 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.462536 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-installation-pull-secrets\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.462811 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.462665 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3f03966a-97af-46f0-99a5-d50a4f54e744-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:12.462889 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.462823 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls podName:699c93ce-4424-4460-8cb2-b5db6ca24033 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:12.962801436 +0000 UTC m=+31.179767237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls") pod "image-registry-68687f4c6c-hd5kb" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033") : secret "image-registry-tls" not found Oct 30 21:42:12.463018 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.463000 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-default-certificate\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.463118 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.459472 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-config\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.463323 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.463301 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81fb1745-4867-438d-b4e0-d44e743e5230-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.463393 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.463325 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:12.463393 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.463375 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert podName:7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:12.96336068 +0000 UTC m=+31.180326480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q9s5" (UID: "7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a") : secret "networking-console-plugin-cert" not found Oct 30 21:42:12.463493 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.463481 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:12.96346916 +0000 UTC m=+31.180434981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:12.463561 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.463517 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-snapshots\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.463614 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.463595 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-trusted-ca\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.463869 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.463814 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/81fb1745-4867-438d-b4e0-d44e743e5230-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.463982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.463938 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a474868-17e7-4ad9-a3f7-65376fc52940-ca-trust-extracted\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.464257 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.464227 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-service-ca-bundle\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.464823 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.464800 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b21892bd-c791-419a-a1d7-4bdc643b7f07-serving-cert\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.465099 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.465064 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:12.469247 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.465817 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6307df15-18fe-45cb-b509-298bcb1fcf3b-cert\") pod \"ingress-canary-trg4h\" (UID: \"6307df15-18fe-45cb-b509-298bcb1fcf3b\") " pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:12.469247 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.466066 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:12.469247 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.466153 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls podName:12730f3a-4d15-47f1-af22-9f4e37eedac7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:12.966136886 +0000 UTC m=+31.183102687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls") pod "dns-default-l8jvk" (UID: "12730f3a-4d15-47f1-af22-9f4e37eedac7") : secret "dns-default-metrics-tls" not found Oct 30 21:42:12.469247 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.466307 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.469247 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.466440 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-installation-pull-secrets\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.469247 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.466562 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-trusted-ca\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.469247 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.467999 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-serving-cert\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.470101 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.470077 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-image-registry-private-configuration\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.476923 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.476892 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qcvmd" Oct 30 21:42:12.482399 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.482380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dq5vk" Oct 30 21:42:12.488701 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.488676 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f47zh" Oct 30 21:42:12.495338 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.495315 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27" Oct 30 21:42:12.515098 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.515065 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:12.518092 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.517596 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cgw\" (UniqueName: \"kubernetes.io/projected/12730f3a-4d15-47f1-af22-9f4e37eedac7-kube-api-access-c9cgw\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.518092 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.517743 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mt5k\" (UniqueName: \"kubernetes.io/projected/3f03966a-97af-46f0-99a5-d50a4f54e744-kube-api-access-8mt5k\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:12.520786 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.520501 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-bound-sa-token\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.520899 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.520828 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pnln\" (UniqueName: \"kubernetes.io/projected/56e7aa71-e129-4ceb-91a9-06770acc40f7-kube-api-access-8pnln\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.521920 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.521875 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbcw8\" (UniqueName: \"kubernetes.io/projected/6307df15-18fe-45cb-b509-298bcb1fcf3b-kube-api-access-tbcw8\") pod \"ingress-canary-trg4h\" (UID: \"6307df15-18fe-45cb-b509-298bcb1fcf3b\") " pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:12.523738 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.523688 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txqg8\" (UniqueName: \"kubernetes.io/projected/65c914e8-45f5-4219-9790-b5fc849d39f7-kube-api-access-txqg8\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:12.524317 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.524273 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvjt\" (UniqueName: \"kubernetes.io/projected/81fb1745-4867-438d-b4e0-d44e743e5230-kube-api-access-4rvjt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.524978 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.524918 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4f9s\" (UniqueName: \"kubernetes.io/projected/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-kube-api-access-j4f9s\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.525304 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.525279 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnrn\" (UniqueName: \"kubernetes.io/projected/5387b573-d7bf-47bc-9d20-755835781b5b-kube-api-access-7bnrn\") pod \"network-check-source-6f78d59f99-l2prc\" (UID: \"5387b573-d7bf-47bc-9d20-755835781b5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc" Oct 30 21:42:12.528110 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.525811 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzqbm\" (UniqueName: \"kubernetes.io/projected/b21892bd-c791-419a-a1d7-4bdc643b7f07-kube-api-access-kzqbm\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.530683 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.530635 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58245\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-kube-api-access-58245\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.531399 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.531358 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7l4k\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-kube-api-access-t7l4k\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.535060 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.533875 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-bound-sa-token\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.537995 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.537921 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:12.577272 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.577124 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:12.599209 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.597501 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:12.605909 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.602321 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc" Oct 30 21:42:12.622678 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.620652 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:12.875982 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.875956 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dq5vk"] Oct 30 21:42:12.880413 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.880387 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f47zh"] Oct 30 21:42:12.883481 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:42:12.883459 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff7cebc_f18e_4b88_bb91_bec7ed9ab097.slice/crio-6c21b8f7a1bdfbae68dc916f3a42a5045081f3dc6a67915dc59d01361ea7c12a WatchSource:0}: Error finding container 6c21b8f7a1bdfbae68dc916f3a42a5045081f3dc6a67915dc59d01361ea7c12a: Status 404 returned error can't find the container with id 6c21b8f7a1bdfbae68dc916f3a42a5045081f3dc6a67915dc59d01361ea7c12a Oct 30 21:42:12.923938 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.923908 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27"] Oct 30 21:42:12.927611 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.927530 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw"] Oct 30 21:42:12.928786 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.928675 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qcvmd"] Oct 30 21:42:12.931874 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:42:12.931844 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22242d39_b0fc_48a0_a9f2_9a28f7106fbd.slice/crio-6b7c5d8b289c438aa7c0887e8172787cd7244bbd78a841f57b10cae4bed7427a WatchSource:0}: Error finding container 6b7c5d8b289c438aa7c0887e8172787cd7244bbd78a841f57b10cae4bed7427a: Status 404 returned error can't find the container with id 6b7c5d8b289c438aa7c0887e8172787cd7244bbd78a841f57b10cae4bed7427a Oct 30 21:42:12.933585 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:42:12.933558 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcc5627_b74e_49ed_a82b_36c2920b9917.slice/crio-d277dc06d694c1abed779199fbc98886281c01de739b0676bb48321205256450 WatchSource:0}: Error finding container d277dc06d694c1abed779199fbc98886281c01de739b0676bb48321205256450: Status 404 returned error can't find the container with id d277dc06d694c1abed779199fbc98886281c01de739b0676bb48321205256450 Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.967388 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.967455 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.967505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.967548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.967626 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.967671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.967707 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.967747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.967878 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.967934 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls podName:12730f3a-4d15-47f1-af22-9f4e37eedac7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.96791756 +0000 UTC m=+32.184883376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls") pod "dns-default-l8jvk" (UID: "12730f3a-4d15-47f1-af22-9f4e37eedac7") : secret "dns-default-metrics-tls" not found Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968022 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968034 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd49c4b-bf2dt: secret "image-registry-tls" not found Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968065 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls podName:8a474868-17e7-4ad9-a3f7-65376fc52940 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.968054733 +0000 UTC m=+32.185020546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls") pod "image-registry-76cdd49c4b-bf2dt" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940") : secret "image-registry-tls" not found Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968115 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968144 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.968134112 +0000 UTC m=+32.185099925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : secret "router-metrics-certs-default" not found Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968197 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:12.968385 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968209 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68687f4c6c-hd5kb: secret "image-registry-tls" not found Oct 30 21:42:12.969243 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968238 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls podName:699c93ce-4424-4460-8cb2-b5db6ca24033 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.968228571 +0000 UTC m=+32.185194384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls") pod "image-registry-68687f4c6c-hd5kb" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033") : secret "image-registry-tls" not found Oct 30 21:42:12.969243 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968243 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:12.969243 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968292 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls podName:3f03966a-97af-46f0-99a5-d50a4f54e744 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.968278461 +0000 UTC m=+32.185244263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xmt2j" (UID: "3f03966a-97af-46f0-99a5-d50a4f54e744") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:12.969243 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968305 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.968299803 +0000 UTC m=+32.185265600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:12.969243 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968327 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:12.969243 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968342 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:12.969243 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968357 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert podName:7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.968347464 +0000 UTC m=+32.185313297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q9s5" (UID: "7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a") : secret "networking-console-plugin-cert" not found Oct 30 21:42:12.969243 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:12.968371 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls podName:65c914e8-45f5-4219-9790-b5fc849d39f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.968364161 +0000 UTC m=+32.185329960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jhjqr" (UID: "65c914e8-45f5-4219-9790-b5fc849d39f7") : secret "samples-operator-tls" not found Oct 30 21:42:12.980054 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.980024 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wbpqf"] Oct 30 21:42:12.985833 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:42:12.985796 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21892bd_c791_419a_a1d7_4bdc643b7f07.slice/crio-80cca2c6f465b9aee7c53bca624627e74f650c28693b6b70f26804479f89109e WatchSource:0}: Error finding container 80cca2c6f465b9aee7c53bca624627e74f650c28693b6b70f26804479f89109e: Status 404 returned error can't find the container with id 80cca2c6f465b9aee7c53bca624627e74f650c28693b6b70f26804479f89109e Oct 30 21:42:12.990474 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:12.989892 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-bthdr"] Oct 30 21:42:12.994795 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:42:12.994770 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d15d9a_71ac_4572_98ca_c5f5df251cb2.slice/crio-dea1b692b04a74ac2f4122ee19064d6d821565a931a3e98b1cdb953c0421fedc WatchSource:0}: Error finding container dea1b692b04a74ac2f4122ee19064d6d821565a931a3e98b1cdb953c0421fedc: Status 404 returned error can't find the container with id dea1b692b04a74ac2f4122ee19064d6d821565a931a3e98b1cdb953c0421fedc Oct 30 21:42:13.010685 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.010653 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc"] Oct 30 21:42:13.015757 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:42:13.015725 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5387b573_d7bf_47bc_9d20_755835781b5b.slice/crio-318e373e1869f1109dc6c47dcbbedf850d454dd8f4fd0c12ac9d92102bd16b4e WatchSource:0}: Error finding container 318e373e1869f1109dc6c47dcbbedf850d454dd8f4fd0c12ac9d92102bd16b4e: Status 404 returned error can't find the container with id 318e373e1869f1109dc6c47dcbbedf850d454dd8f4fd0c12ac9d92102bd16b4e Oct 30 21:42:13.018399 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.017464 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks"] Oct 30 21:42:13.021732 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:42:13.021698 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fb1745_4867_438d_b4e0_d44e743e5230.slice/crio-60fbc58e6170538cfe71c93892d3bcc81b0c1d70980153533671ac93b0325ad9 WatchSource:0}: Error finding container 60fbc58e6170538cfe71c93892d3bcc81b0c1d70980153533671ac93b0325ad9: Status 404 returned error can't find the container with id 60fbc58e6170538cfe71c93892d3bcc81b0c1d70980153533671ac93b0325ad9 Oct 30 21:42:13.026791 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.026425 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-trg4h"] Oct 30 21:42:13.030312 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:42:13.030290 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6307df15_18fe_45cb_b509_298bcb1fcf3b.slice/crio-04ac54ea25c108341d4c55f6c2444f650cfc4531266e7ef94337cfd4d29c503e WatchSource:0}: Error finding container 04ac54ea25c108341d4c55f6c2444f650cfc4531266e7ef94337cfd4d29c503e: Status 404 returned error can't find the container with id 04ac54ea25c108341d4c55f6c2444f650cfc4531266e7ef94337cfd4d29c503e Oct 30 21:42:13.265938 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.265907 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:13.266091 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.265969 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:13.318285 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.318210 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:42:13.318285 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.318267 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9csh8\"" Oct 30 21:42:13.318871 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.318320 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sflz7\"" Oct 30 21:42:13.474389 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.474358 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" event={"ID":"81fb1745-4867-438d-b4e0-d44e743e5230","Type":"ContainerStarted","Data":"60fbc58e6170538cfe71c93892d3bcc81b0c1d70980153533671ac93b0325ad9"} Oct 30 21:42:13.475528 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.475503 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc" event={"ID":"5387b573-d7bf-47bc-9d20-755835781b5b","Type":"ContainerStarted","Data":"318e373e1869f1109dc6c47dcbbedf850d454dd8f4fd0c12ac9d92102bd16b4e"} Oct 30 21:42:13.476537 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.476517 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27" event={"ID":"22242d39-b0fc-48a0-a9f2-9a28f7106fbd","Type":"ContainerStarted","Data":"6b7c5d8b289c438aa7c0887e8172787cd7244bbd78a841f57b10cae4bed7427a"} Oct 30 21:42:13.477892 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.477872 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f47zh" event={"ID":"aff7cebc-f18e-4b88-bb91-bec7ed9ab097","Type":"ContainerStarted","Data":"866b1f1fdf8db58341a8007098b1ce51f9ba2b7becd4f4184acd27aabbc32984"} Oct 30 21:42:13.478003 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.477907 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f47zh" event={"ID":"aff7cebc-f18e-4b88-bb91-bec7ed9ab097","Type":"ContainerStarted","Data":"6c21b8f7a1bdfbae68dc916f3a42a5045081f3dc6a67915dc59d01361ea7c12a"} Oct 30 21:42:13.482832 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.482808 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-trg4h" event={"ID":"6307df15-18fe-45cb-b509-298bcb1fcf3b","Type":"ContainerStarted","Data":"04ac54ea25c108341d4c55f6c2444f650cfc4531266e7ef94337cfd4d29c503e"} Oct 30 21:42:13.483822 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.483804 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wbpqf" event={"ID":"b21892bd-c791-419a-a1d7-4bdc643b7f07","Type":"ContainerStarted","Data":"80cca2c6f465b9aee7c53bca624627e74f650c28693b6b70f26804479f89109e"} Oct 30 21:42:13.484836 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.484816 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" event={"ID":"f9d15d9a-71ac-4572-98ca-c5f5df251cb2","Type":"ContainerStarted","Data":"dea1b692b04a74ac2f4122ee19064d6d821565a931a3e98b1cdb953c0421fedc"} Oct 30 21:42:13.485844 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.485824 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" event={"ID":"3bcc5627-b74e-49ed-a82b-36c2920b9917","Type":"ContainerStarted","Data":"d277dc06d694c1abed779199fbc98886281c01de739b0676bb48321205256450"} Oct 30 21:42:13.487125 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.487099 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dq5vk" event={"ID":"5197a548-5e64-49a3-a5b4-421898abe7df","Type":"ContainerStarted","Data":"6e2217a7711953ba9fb120186aae0a1e47ba485b3ed0cf0fb8044539ef7e33ae"} Oct 30 21:42:13.487197 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.487147 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dq5vk" event={"ID":"5197a548-5e64-49a3-a5b4-421898abe7df","Type":"ContainerStarted","Data":"b8377cf2a5da96a735d8ed758cc5bf91d3d741598a266d8073a391bbd14170da"} Oct 30 21:42:13.488434 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.488417 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qcvmd" event={"ID":"78532fdc-d35e-4ec3-a48a-18fe0bbe8484","Type":"ContainerStarted","Data":"cadf4a996fd3f61cc741e719008d0a81e3ea4b630851fe914867c325878ec4bc"} Oct 30 21:42:13.488496 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.488446 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qcvmd" event={"ID":"78532fdc-d35e-4ec3-a48a-18fe0bbe8484","Type":"ContainerStarted","Data":"2066f57b222fe1f6eb5d43dfcc127adf368a1423896d6f5bc4e77b24d53f18a8"} Oct 30 21:42:13.976593 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.976557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:13.976783 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.976625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:13.976783 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.976667 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:13.976783 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.976713 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:13.976783 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.976739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:13.976783 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.976768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:13.977058 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.976839 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:13.977058 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:13.976882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:13.977058 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977035 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.977017345 +0000 UTC m=+34.193983150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:13.977209 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977119 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:13.977209 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977157 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert podName:7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.977145318 +0000 UTC m=+34.194111121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q9s5" (UID: "7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a") : secret "networking-console-plugin-cert" not found Oct 30 21:42:13.977312 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977213 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:13.977312 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977242 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls podName:12730f3a-4d15-47f1-af22-9f4e37eedac7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.977232238 +0000 UTC m=+34.194198042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls") pod "dns-default-l8jvk" (UID: "12730f3a-4d15-47f1-af22-9f4e37eedac7") : secret "dns-default-metrics-tls" not found Oct 30 21:42:13.977312 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977306 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:13.977446 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977316 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd49c4b-bf2dt: secret "image-registry-tls" not found Oct 30 21:42:13.977446 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977348 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls podName:8a474868-17e7-4ad9-a3f7-65376fc52940 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.977337342 +0000 UTC m=+34.194303145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls") pod "image-registry-76cdd49c4b-bf2dt" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940") : secret "image-registry-tls" not found Oct 30 21:42:13.977446 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977405 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:13.977596 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977525 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:13.977596 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977560 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:13.977683 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977667 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:13.977683 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977678 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68687f4c6c-hd5kb: secret "image-registry-tls" not found Oct 30 21:42:13.977774 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977727 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.977423886 +0000 UTC m=+34.194389686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : secret "router-metrics-certs-default" not found Oct 30 21:42:13.977774 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977745 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls podName:65c914e8-45f5-4219-9790-b5fc849d39f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.97773563 +0000 UTC m=+34.194701433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jhjqr" (UID: "65c914e8-45f5-4219-9790-b5fc849d39f7") : secret "samples-operator-tls" not found Oct 30 21:42:13.977774 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977760 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls podName:3f03966a-97af-46f0-99a5-d50a4f54e744 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.977751703 +0000 UTC m=+34.194717508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xmt2j" (UID: "3f03966a-97af-46f0-99a5-d50a4f54e744") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:13.977932 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:13.977776 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls podName:699c93ce-4424-4460-8cb2-b5db6ca24033 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.9777683 +0000 UTC m=+34.194734100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls") pod "image-registry-68687f4c6c-hd5kb" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033") : secret "image-registry-tls" not found Oct 30 21:42:14.310136 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:14.308374 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qcvmd" podStartSLOduration=373.308356228 podStartE2EDuration="6m13.308356228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:14.307531829 +0000 UTC m=+32.524497650" watchObservedRunningTime="2026-10-30 21:42:14.308356228 +0000 UTC m=+32.525322050" Oct 30 21:42:14.429565 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:14.429045 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dq5vk" podStartSLOduration=373.429025467 podStartE2EDuration="6m13.429025467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:14.427329476 +0000 UTC m=+32.644295302" watchObservedRunningTime="2026-10-30 21:42:14.429025467 +0000 UTC m=+32.645991288" Oct 30 21:42:14.539637 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:14.539572 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-f47zh" podStartSLOduration=373.53955778 podStartE2EDuration="6m13.53955778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:14.538931155 +0000 UTC m=+32.755896976" watchObservedRunningTime="2026-10-30 21:42:14.53955778 +0000 UTC m=+32.756523603" Oct 30 21:42:15.192063 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:15.192027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:15.213535 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:15.213508 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:15.283682 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:15.283645 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:15.294392 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:15.293755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:15.294392 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:15.293805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:15.294392 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:15.293988 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:42:15.294392 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:15.294127 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:47.29409042 +0000 UTC m=+65.511056236 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : secret "metrics-daemon-secret" not found Oct 30 21:42:15.303023 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:15.302965 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:15.378541 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:15.377221 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:15.691658 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:15.691626 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z258k"] Oct 30 21:42:15.800868 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:15.800790 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jnj6f"] Oct 30 21:42:15.810353 ip-10-0-141-104 kubenswrapper[2581]: W1030 21:42:15.810310 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda403f7b4_a616_4807_8263_678045389f84.slice/crio-d89e4ed6631fbd05fe2eef9e0c60ef3712d511f053ec7fa47506593fa3e7ace2 WatchSource:0}: Error finding container d89e4ed6631fbd05fe2eef9e0c60ef3712d511f053ec7fa47506593fa3e7ace2: Status 404 returned error can't find the container with id d89e4ed6631fbd05fe2eef9e0c60ef3712d511f053ec7fa47506593fa3e7ace2 Oct 30 21:42:16.003740 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.003656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:16.003740 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.003702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:16.003740 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.003736 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:16.004204 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.003785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:16.004204 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.003829 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:16.004204 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.003865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:16.004204 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.003902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:16.004204 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.003958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:16.004204 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004105 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:16.004204 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004121 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd49c4b-bf2dt: secret "image-registry-tls" not found Oct 30 21:42:16.004204 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004177 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls podName:8a474868-17e7-4ad9-a3f7-65376fc52940 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.004159642 +0000 UTC m=+38.221125456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls") pod "image-registry-76cdd49c4b-bf2dt" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940") : secret "image-registry-tls" not found Oct 30 21:42:16.004691 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004571 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:16.004691 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004620 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.004606211 +0000 UTC m=+38.221572022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : secret "router-metrics-certs-default" not found Oct 30 21:42:16.004691 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004683 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:16.004879 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004694 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68687f4c6c-hd5kb: secret "image-registry-tls" not found Oct 30 21:42:16.004879 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004729 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls podName:699c93ce-4424-4460-8cb2-b5db6ca24033 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.004717768 +0000 UTC m=+38.221683580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls") pod "image-registry-68687f4c6c-hd5kb" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033") : secret "image-registry-tls" not found Oct 30 21:42:16.004879 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004786 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:16.004879 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004821 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls podName:65c914e8-45f5-4219-9790-b5fc849d39f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.004810792 +0000 UTC m=+38.221776603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jhjqr" (UID: "65c914e8-45f5-4219-9790-b5fc849d39f7") : secret "samples-operator-tls" not found Oct 30 21:42:16.004879 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004872 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:16.005269 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004903 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls podName:3f03966a-97af-46f0-99a5-d50a4f54e744 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.004893391 +0000 UTC m=+38.221859203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xmt2j" (UID: "3f03966a-97af-46f0-99a5-d50a4f54e744") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:16.005269 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.004985 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.004974969 +0000 UTC m=+38.221940782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:16.005269 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.005040 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:16.005269 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.005074 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert podName:7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.005064216 +0000 UTC m=+38.222030029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q9s5" (UID: "7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a") : secret "networking-console-plugin-cert" not found Oct 30 21:42:16.005269 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.005126 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:16.005269 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:16.005155 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls podName:12730f3a-4d15-47f1-af22-9f4e37eedac7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.005145418 +0000 UTC m=+38.222111230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls") pod "dns-default-l8jvk" (UID: "12730f3a-4d15-47f1-af22-9f4e37eedac7") : secret "dns-default-metrics-tls" not found Oct 30 21:42:16.510527 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.510390 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jnj6f" event={"ID":"a403f7b4-a616-4807-8263-678045389f84","Type":"ContainerStarted","Data":"d89e4ed6631fbd05fe2eef9e0c60ef3712d511f053ec7fa47506593fa3e7ace2"} Oct 30 21:42:16.513338 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.512565 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27" event={"ID":"22242d39-b0fc-48a0-a9f2-9a28f7106fbd","Type":"ContainerStarted","Data":"390ae90b73a3e9f2b46ed537d815c82f1eab0fb73ba28e78ea25234b214d76ea"} Oct 30 21:42:16.515415 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.515346 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z258k" event={"ID":"70fbe0ee-70e4-47b1-8094-4faa96292c6b","Type":"ContainerStarted","Data":"b34d7d0d5ccdeacfd4672d95203f4194409a25f2597a325575c02d3ce076fb9a"} Oct 30 21:42:16.628280 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:16.628231 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27" podStartSLOduration=333.904235813 podStartE2EDuration="5m36.628217041s" totalImagesPullingTime="2.723981228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:16.626331281 +0000 UTC m=+34.843297103" watchObservedRunningTime="2026-10-30 21:42:16.628217041 +0000 UTC m=+34.845182860" Oct 30 21:42:17.522961 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:17.522923 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/0.log" Oct 30 21:42:17.523323 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:17.522978 2581 generic.go:356] "Generic (PLEG): container finished" podID="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" containerID="8745f278f188af0f5a335fdfb34726c855554946c442d1fdc20f5423329513b1" exitCode=255 Oct 30 21:42:17.523673 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:17.523658 2581 scope.go:122] "RemoveContainer" containerID="8745f278f188af0f5a335fdfb34726c855554946c442d1fdc20f5423329513b1" Oct 30 21:42:17.523896 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:17.523847 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" event={"ID":"f9d15d9a-71ac-4572-98ca-c5f5df251cb2","Type":"ContainerDied","Data":"8745f278f188af0f5a335fdfb34726c855554946c442d1fdc20f5423329513b1"} Oct 30 21:42:18.529998 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.529323 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wbpqf" event={"ID":"b21892bd-c791-419a-a1d7-4bdc643b7f07","Type":"ContainerStarted","Data":"939774b2ed2eb204240d907f6fe10fd742dcc74882930720063c773ffe191678"} Oct 30 21:42:18.544964 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.532524 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/1.log" Oct 30 21:42:18.544964 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.533171 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/0.log" Oct 30 21:42:18.544964 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.533194 2581 generic.go:356] "Generic (PLEG): container finished" podID="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" containerID="328a8eaab798437384f355cbcbec523ecb9f13793741b3dada3d785415388da3" exitCode=255 Oct 30 21:42:18.544964 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.533252 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" event={"ID":"f9d15d9a-71ac-4572-98ca-c5f5df251cb2","Type":"ContainerDied","Data":"328a8eaab798437384f355cbcbec523ecb9f13793741b3dada3d785415388da3"} Oct 30 21:42:18.544964 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.533289 2581 scope.go:122] "RemoveContainer" containerID="8745f278f188af0f5a335fdfb34726c855554946c442d1fdc20f5423329513b1" Oct 30 21:42:18.544964 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.533601 2581 scope.go:122] "RemoveContainer" containerID="328a8eaab798437384f355cbcbec523ecb9f13793741b3dada3d785415388da3" Oct 30 21:42:18.544964 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:18.533770 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthdr_openshift-console-operator(f9d15d9a-71ac-4572-98ca-c5f5df251cb2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" podUID="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" Oct 30 21:42:18.544964 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.537209 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" event={"ID":"3bcc5627-b74e-49ed-a82b-36c2920b9917","Type":"ContainerStarted","Data":"fe760cab44c7134140a0afbf88ad15c5abab6a5e91e4ad50a50af579a2cf4cdb"} Oct 30 21:42:18.544964 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.540740 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" event={"ID":"81fb1745-4867-438d-b4e0-d44e743e5230","Type":"ContainerStarted","Data":"2dabd9fc5fbc8c6697d0cdca6e615a8052539d688b52b22d038125154d6789bd"} Oct 30 21:42:18.659075 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.658803 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-wbpqf" podStartSLOduration=333.196741158 podStartE2EDuration="5m37.658784005s" totalImagesPullingTime="4.462042847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:18.656970815 +0000 UTC m=+36.873936636" watchObservedRunningTime="2026-10-30 21:42:18.658784005 +0000 UTC m=+36.875749826" Oct 30 21:42:18.772566 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.772505 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" podStartSLOduration=337.554315278 podStartE2EDuration="5m42.772484745s" totalImagesPullingTime="5.218169467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:18.772219822 +0000 UTC m=+36.989185643" watchObservedRunningTime="2026-10-30 21:42:18.772484745 +0000 UTC m=+36.989450568" Oct 30 21:42:18.854622 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.854327 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r246j"] Oct 30 21:42:18.868042 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.868013 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:18.919139 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.917644 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r246j"] Oct 30 21:42:18.920654 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.920120 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:42:18.920791 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.920665 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:42:18.921035 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:18.920895 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wcr9d\"" Oct 30 21:42:19.002576 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.002512 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" podStartSLOduration=335.05831821 podStartE2EDuration="5m40.002491608s" totalImagesPullingTime="4.944173398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:19.000703356 +0000 UTC m=+37.217669208" watchObservedRunningTime="2026-10-30 21:42:19.002491608 +0000 UTC m=+37.219457428" Oct 30 21:42:19.037004 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.035121 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-crio-socket\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.037004 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.035190 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.037004 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.035311 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.037004 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.035378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45k86\" (UniqueName: \"kubernetes.io/projected/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-api-access-45k86\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.037004 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.035433 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-data-volume\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.135999 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.135893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.135999 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.135976 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45k86\" (UniqueName: \"kubernetes.io/projected/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-api-access-45k86\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.136198 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.136015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-data-volume\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.136198 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.136139 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-crio-socket\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.136198 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.136176 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.136340 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:19.136294 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:19.136396 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:19.136351 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.636334071 +0000 UTC m=+37.853299873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:19.136635 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.136613 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-crio-socket\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.136708 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:19.136689 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:19.136751 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:19.136729 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.63671309 +0000 UTC m=+37.853678893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : configmap "kube-rbac-proxy" not found Oct 30 21:42:19.136751 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.136735 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-data-volume\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.199936 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.199903 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45k86\" (UniqueName: \"kubernetes.io/projected/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-api-access-45k86\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.546449 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.546350 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-trg4h" event={"ID":"6307df15-18fe-45cb-b509-298bcb1fcf3b","Type":"ContainerStarted","Data":"1caf379352c2f79d08f99fd2158257a20eadbe9d734265c71a6e7d1b7b157476"} Oct 30 21:42:19.548102 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.548077 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/1.log" Oct 30 21:42:19.548457 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.548437 2581 scope.go:122] "RemoveContainer" containerID="328a8eaab798437384f355cbcbec523ecb9f13793741b3dada3d785415388da3" Oct 30 21:42:19.548778 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:19.548752 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthdr_openshift-console-operator(f9d15d9a-71ac-4572-98ca-c5f5df251cb2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" podUID="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" Oct 30 21:42:19.643458 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.641978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.643458 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.642062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:19.643458 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:19.642142 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:19.643458 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:19.642205 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.64218589 +0000 UTC m=+38.859151705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:19.643458 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:19.642226 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:19.643458 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:19.642266 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.642252753 +0000 UTC m=+38.859218567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : configmap "kube-rbac-proxy" not found Oct 30 21:42:19.661498 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:19.661450 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-trg4h" podStartSLOduration=2.092383983 podStartE2EDuration="7.66143622s" totalImagesPullingTime="5.569052237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:19.660379294 +0000 UTC m=+37.877345115" watchObservedRunningTime="2026-10-30 21:42:19.66143622 +0000 UTC m=+37.878402050" Oct 30 21:42:20.045752 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.045717 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:20.045925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.045768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:20.045925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.045822 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:20.045925 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.045857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:20.045925 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.045910 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.045885159 +0000 UTC m=+46.262850960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.045976 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.045971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.045989 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd49c4b-bf2dt: secret "image-registry-tls" not found Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.046020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046040 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls podName:8a474868-17e7-4ad9-a3f7-65376fc52940 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.046027619 +0000 UTC m=+46.262993421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls") pod "image-registry-76cdd49c4b-bf2dt" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940") : secret "image-registry-tls" not found Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.046065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046103 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046115 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68687f4c6c-hd5kb: secret "image-registry-tls" not found Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046145 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046149 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls podName:699c93ce-4424-4460-8cb2-b5db6ca24033 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.046138682 +0000 UTC m=+46.263104483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls") pod "image-registry-68687f4c6c-hd5kb" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033") : secret "image-registry-tls" not found Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.046103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046169 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls podName:3f03966a-97af-46f0-99a5-d50a4f54e744 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.046161563 +0000 UTC m=+46.263127361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xmt2j" (UID: "3f03966a-97af-46f0-99a5-d50a4f54e744") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046249 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:20.046300 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046282 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert podName:7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.046272628 +0000 UTC m=+46.263238428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q9s5" (UID: "7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a") : secret "networking-console-plugin-cert" not found Oct 30 21:42:20.046851 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046326 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:20.046851 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046346 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls podName:12730f3a-4d15-47f1-af22-9f4e37eedac7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.046339993 +0000 UTC m=+46.263305790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls") pod "dns-default-l8jvk" (UID: "12730f3a-4d15-47f1-af22-9f4e37eedac7") : secret "dns-default-metrics-tls" not found Oct 30 21:42:20.046851 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046377 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:20.046851 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046395 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls podName:65c914e8-45f5-4219-9790-b5fc849d39f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.046389346 +0000 UTC m=+46.263355143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jhjqr" (UID: "65c914e8-45f5-4219-9790-b5fc849d39f7") : secret "samples-operator-tls" not found Oct 30 21:42:20.046851 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046430 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:20.046851 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.046450 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.046443554 +0000 UTC m=+46.263409352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : secret "router-metrics-certs-default" not found Oct 30 21:42:20.553823 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.553364 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc" event={"ID":"5387b573-d7bf-47bc-9d20-755835781b5b","Type":"ContainerStarted","Data":"b2ad2e287fd44fff24189005e22f551c3625676485851c2252ba142640477ad9"} Oct 30 21:42:20.555139 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.555111 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jnj6f" event={"ID":"a403f7b4-a616-4807-8263-678045389f84","Type":"ContainerStarted","Data":"ac8b9a8c9678438bc41612b6ba33aa340cfabd0cb4e6738f1692a218f71f10a7"} Oct 30 21:42:20.555417 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.555401 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:20.652172 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.652136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:20.652172 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.652179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:20.653640 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.653615 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:20.653735 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.653678 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.65366058 +0000 UTC m=+40.870626379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:20.653735 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.653714 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:20.653835 ip-10-0-141-104 kubenswrapper[2581]: E1030 21:42:20.653745 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.653735979 +0000 UTC m=+40.870701779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : configmap "kube-rbac-proxy" not found Oct 30 21:42:20.667705 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.667636 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc" podStartSLOduration=331.134766582 podStartE2EDuration="5m37.667619097s" totalImagesPullingTime="6.532852515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:20.666350133 +0000 UTC m=+38.883315953" watchObservedRunningTime="2026-10-30 21:42:20.667619097 +0000 UTC m=+38.884584916" Oct 30 21:42:20.779517 ip-10-0-141-104 kubenswrapper[2581]: I1030 21:42:20.779452 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jnj6f" podStartSLOduration=33.312529779 podStartE2EDuration="37.779435444s" totalImagesPullingTime="4.466905665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:20.778179777 +0000 UTC m=+38.995145598" watchObservedRunningTime="2026-10-30 21:42:20.779435444 +0000 UTC m=+38.996401265" Oct 30 21:42:22.559609 ip-10-0-141-104 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:42:22.575266 ip-10-0-141-104 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:42:22.575481 ip-10-0-141-104 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:42:22.575692 ip-10-0-141-104 systemd[1]: kubelet.service: Consumed 2.868s CPU time. Oct 30 21:42:22.583500 ip-10-0-141-104 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:42:22.659987 ip-10-0-141-104 kubenswrapper[4859]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:42:22.659987 ip-10-0-141-104 kubenswrapper[4859]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:42:22.659987 ip-10-0-141-104 kubenswrapper[4859]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:42:22.659987 ip-10-0-141-104 kubenswrapper[4859]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:42:22.662054 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.661969 4859 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:22.662054 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662053 4859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:22.662054 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662057 4859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662061 4859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662064 4859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662067 4859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662070 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662079 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662082 4859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662085 4859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662088 4859 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662091 4859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662094 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662097 4859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662099 4859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662102 4859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662104 4859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662107 4859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662110 4859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662113 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662116 4859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:22.662154 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662119 4859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662121 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662124 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662127 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662129 4859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662132 4859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662134 4859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662137 4859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662140 4859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662143 4859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662146 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662149 4859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662152 4859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662154 4859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662157 4859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662160 4859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662165 4859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662170 4859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662172 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:22.662625 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662175 4859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662178 4859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662180 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662183 4859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662186 4859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662188 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662191 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662193 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662196 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662199 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662202 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662205 4859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662207 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662210 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662213 4859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662215 4859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662218 4859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662220 4859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662223 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:22.663168 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662225 4859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662228 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662230 4859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662233 4859 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662235 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662238 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662247 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662250 4859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662253 4859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662255 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662258 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662261 4859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662263 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662266 4859 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662270 4859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662274 4859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662277 4859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662280 4859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662283 4859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:22.663641 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662285 4859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662288 4859 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662291 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662293 4859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662296 4859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662298 4859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662301 4859 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662303 4859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662306 4859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662309 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662312 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662314 4859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662317 4859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662320 4859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662322 4859 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662325 4859 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662328 4859 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662330 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662333 4859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662335 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:22.664129 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662343 4859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662346 4859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662349 4859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662352 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662355 4859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662358 4859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662361 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.662364 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663162 4859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663171 4859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663175 4859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663178 4859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663182 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663185 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663188 4859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663191 4859 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663193 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663196 4859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663199 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663203 4859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:22.664616 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663207 4859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663210 4859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663213 4859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663215 4859 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663218 4859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663220 4859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663223 4859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663226 4859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663228 4859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663231 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663234 4859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663236 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663239 4859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663242 4859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663245 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663248 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663250 4859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663253 4859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663255 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663258 4859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:22.665121 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663260 4859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663263 4859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663265 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663268 4859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663270 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663273 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663275 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663278 4859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663281 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663284 4859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663287 4859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663289 4859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663292 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663294 4859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663297 4859 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663299 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663302 4859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663304 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663307 4859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:22.665613 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663309 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663312 4859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663314 4859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663317 4859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663319 4859 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663322 4859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663326 4859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663331 4859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663335 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663337 4859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663340 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663343 4859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663346 4859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663349 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663351 4859 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663354 4859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663357 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663359 4859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663362 4859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663364 4859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:22.666111 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663367 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663369 4859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663372 4859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663375 4859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663377 4859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663380 4859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663382 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663385 4859 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663387 4859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663390 4859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663393 4859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663395 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663398 4859 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663400 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663403 4859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663406 4859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663408 4859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663411 4859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663413 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663416 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:22.666601 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663420 4859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663422 4859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663425 4859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663428 4859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663431 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663433 4859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663437 4859 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663440 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663442 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663445 4859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663447 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663450 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663452 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663454 4859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663457 4859 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663930 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663935 4859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663937 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663955 4859 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:22.667127 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663959 4859 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663962 4859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663965 4859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663968 4859 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663971 4859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663973 4859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663976 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663979 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663982 4859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663984 4859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663987 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663990 4859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663993 4859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.663997 4859 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664001 4859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664005 4859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664007 4859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664010 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664012 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:22.667610 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664015 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664017 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664020 4859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664022 4859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664025 4859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664028 4859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664031 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664034 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664036 4859 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664039 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664041 4859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664044 4859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664047 4859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664049 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664052 4859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664054 4859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664057 4859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664060 4859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664062 4859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:22.668189 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664065 4859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664067 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664070 4859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664072 4859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664075 4859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664077 4859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664080 4859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664082 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664085 4859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664088 4859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664090 4859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664093 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664095 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664098 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664100 4859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664103 4859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664105 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664108 4859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664110 4859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664113 4859 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:22.668973 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664115 4859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664118 4859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664120 4859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664123 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664125 4859 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664127 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664130 4859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664133 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664136 4859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664139 4859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664141 4859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664144 4859 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664147 4859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664149 4859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664152 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664155 4859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664157 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664160 4859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664162 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664165 4859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:22.669549 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664167 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664171 4859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664173 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664176 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664178 4859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664181 4859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664184 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664186 4859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664190 4859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664193 4859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664196 4859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664203 4859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664206 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664209 4859 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664212 4859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664214 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664217 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664219 4859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664222 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:22.670098 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664224 4859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664226 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664230 4859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664233 4859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.664235 4859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664327 4859 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664334 4859 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664340 4859 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664345 4859 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664349 4859 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664352 4859 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664357 4859 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664363 4859 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664366 4859 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664369 4859 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664373 4859 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664377 4859 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664380 4859 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664383 4859 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664386 4859 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664389 4859 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664392 4859 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664395 4859 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664399 4859 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:42:22.670569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664404 4859 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664408 4859 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664411 4859 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664414 4859 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664418 4859 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664421 4859 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664424 4859 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664427 4859 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664431 4859 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664434 4859 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664437 4859 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664440 4859 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664443 4859 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664447 4859 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664450 4859 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664454 4859 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664457 4859 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664460 4859 flags.go:64] FLAG: --enable-server="true" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664463 4859 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664466 4859 flags.go:64] FLAG: --event-burst="100" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664471 4859 flags.go:64] FLAG: --event-qps="50" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664474 4859 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664478 4859 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664481 4859 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:42:22.671208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664485 4859 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664489 4859 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664492 4859 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664495 4859 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664498 4859 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664501 4859 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664504 4859 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664507 4859 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664512 4859 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664515 4859 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664518 4859 flags.go:64] FLAG: --feature-gates="" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664522 4859 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664525 4859 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664528 4859 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664532 4859 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664535 4859 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664538 4859 flags.go:64] FLAG: --help="false" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664541 4859 flags.go:64] FLAG: --hostname-override="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664544 4859 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664547 4859 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664550 4859 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664554 4859 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664557 4859 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:42:22.671800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664560 4859 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664564 4859 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664566 4859 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664569 4859 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664572 4859 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664577 4859 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664580 4859 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664583 4859 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664586 4859 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664589 4859 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664592 4859 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664595 4859 flags.go:64] FLAG: --lock-file="" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664598 4859 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664601 4859 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664604 4859 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664610 4859 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664613 4859 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664617 4859 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664620 4859 flags.go:64] FLAG: --logging-format="text" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664624 4859 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664627 4859 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664630 4859 flags.go:64] FLAG: --manifest-url="" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664633 4859 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664638 4859 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664641 4859 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:42:22.672378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664645 4859 flags.go:64] FLAG: --max-pods="110" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664648 4859 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664651 4859 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664654 4859 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664657 4859 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664660 4859 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664663 4859 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664665 4859 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664672 4859 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664675 4859 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664678 4859 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664682 4859 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664686 4859 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664689 4859 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664692 4859 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664696 4859 flags.go:64] FLAG: --port="10250" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664699 4859 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664702 4859 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-08f140ac732cf41ed" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664706 4859 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664709 4859 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664713 4859 flags.go:64] FLAG: --register-node="true" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664716 4859 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664719 4859 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664722 4859 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664726 4859 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664729 4859 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:42:22.672997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664733 4859 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664736 4859 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664739 4859 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664742 4859 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664745 4859 flags.go:64] FLAG: --runonce="false" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664748 4859 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664751 4859 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664754 4859 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664757 4859 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664760 4859 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664763 4859 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664766 4859 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664769 4859 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664772 4859 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664774 4859 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664777 4859 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664780 4859 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664783 4859 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664787 4859 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664790 4859 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664796 4859 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664799 4859 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664802 4859 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664809 4859 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664811 4859 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:42:22.673633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664815 4859 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664818 4859 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664821 4859 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664824 4859 flags.go:64] FLAG: --v="2" Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664828 4859 flags.go:64] FLAG: --version="false" Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664832 4859 flags.go:64] FLAG: --vmodule="" Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664836 4859 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.664839 4859 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665012 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665017 4859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665020 4859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665023 4859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665026 4859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665029 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665031 4859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665034 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665036 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665039 4859 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665041 4859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665044 4859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665047 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:22.674286 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665049 4859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665052 4859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665055 4859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665058 4859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665062 4859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665064 4859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665067 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665069 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665072 4859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665076 4859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665080 4859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665083 4859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665086 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665090 4859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665093 4859 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665095 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665098 4859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665100 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665103 4859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:22.674805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665106 4859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665108 4859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665110 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665113 4859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665115 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665118 4859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665121 4859 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665123 4859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665126 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665129 4859 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665132 4859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665134 4859 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665137 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665140 4859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665142 4859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665145 4859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665148 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665153 4859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665156 4859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665159 4859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:22.675593 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665161 4859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665164 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665167 4859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665169 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665172 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665174 4859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665178 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665180 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665183 4859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665186 4859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665188 4859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665191 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665193 4859 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665196 4859 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665198 4859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665201 4859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665203 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665206 4859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665209 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:22.676264 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665211 4859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665214 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665216 4859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665218 4859 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665221 4859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665224 4859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665226 4859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665229 4859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665231 4859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665234 4859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665239 4859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665241 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665244 4859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665246 4859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665249 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665251 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665254 4859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665256 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665259 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665261 4859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:22.676796 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665264 4859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665267 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665272 4859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665275 4859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665278 4859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665281 4859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665284 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665286 4859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665289 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665292 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665294 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665297 4859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665300 4859 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665303 4859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665305 4859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:22.677445 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.665331 4859 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665371 4859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665375 4859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665378 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665381 4859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665384 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665386 4859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665390 4859 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665396 4859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665399 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665402 4859 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665405 4859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665408 4859 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665411 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665413 4859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665416 4859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665418 4859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665421 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665425 4859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665427 4859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:22.677819 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665430 4859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665433 4859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665435 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665438 4859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665441 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665443 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665445 4859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665448 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665451 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665453 4859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665456 4859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665459 4859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665461 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665464 4859 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665466 4859 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665469 4859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665472 4859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665474 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665477 4859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:22.678357 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665481 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665485 4859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665488 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665490 4859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665493 4859 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665496 4859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665498 4859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665501 4859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665503 4859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665506 4859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665508 4859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665511 4859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665514 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665516 4859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665519 4859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665522 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665524 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665527 4859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665529 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665531 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:22.678861 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665534 4859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665536 4859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665539 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665543 4859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665548 4859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665551 4859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665554 4859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665556 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665559 4859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665561 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665563 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665566 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665570 4859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665574 4859 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665577 4859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665579 4859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665582 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665584 4859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665587 4859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:22.679412 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665590 4859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665592 4859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665595 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665597 4859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665600 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665602 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665606 4859 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665608 4859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665611 4859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665614 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665616 4859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665619 4859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665622 4859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665625 4859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665627 4859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665630 4859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665632 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665635 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665638 4859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:22.679870 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665642 4859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665646 4859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665648 4859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665651 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665653 4859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665656 4859 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665660 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665664 4859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665667 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.665669 4859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.670633 4859 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.670656 4859 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.670674 4859 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.670678 4859 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670743 4859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670749 4859 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670752 4859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670755 4859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670758 4859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:22.680352 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670761 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670764 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670766 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670769 4859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670772 4859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670775 4859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670777 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670780 4859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670782 4859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670786 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670788 4859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670791 4859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670794 4859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670796 4859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670799 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670801 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670804 4859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670807 4859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670809 4859 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:22.680813 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670812 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670815 4859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670818 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670822 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670824 4859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670827 4859 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670831 4859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670835 4859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670838 4859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670841 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670844 4859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670847 4859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670849 4859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670852 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670854 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670857 4859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670859 4859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670862 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670864 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:22.681289 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670867 4859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670870 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670872 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670880 4859 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670884 4859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670886 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670889 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670891 4859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670894 4859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670896 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670899 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670903 4859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670906 4859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670909 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670912 4859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670915 4859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670918 4859 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670921 4859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670924 4859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670927 4859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:22.681741 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670930 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670932 4859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670935 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670937 4859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670940 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670958 4859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670961 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670964 4859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670967 4859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670969 4859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670972 4859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670974 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670977 4859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670980 4859 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670982 4859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670985 4859 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670989 4859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670991 4859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670994 4859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670997 4859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:22.682280 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.670999 4859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671002 4859 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671005 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671007 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671010 4859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671013 4859 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671016 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671018 4859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671021 4859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671024 4859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671026 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671029 4859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671032 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671034 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671037 4859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671040 4859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671042 4859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671045 4859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671048 4859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:22.682761 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671050 4859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671053 4859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671056 4859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671059 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.671088 4859 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671250 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671255 4859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671259 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671261 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671264 4859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671268 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671271 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671274 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671276 4859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671279 4859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671282 4859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:22.683244 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671284 4859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671287 4859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671289 4859 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671292 4859 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671295 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671298 4859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671301 4859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671303 4859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671306 4859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671308 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671311 4859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671314 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671316 4859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671319 4859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671322 4859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671325 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671328 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671331 4859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671333 4859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671336 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:22.683765 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671338 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671341 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671343 4859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671346 4859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671349 4859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671351 4859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671353 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671361 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671364 4859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671366 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671369 4859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671371 4859 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671374 4859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671376 4859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671379 4859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671381 4859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671384 4859 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671386 4859 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671389 4859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671391 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:22.684536 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671394 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671397 4859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671399 4859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671402 4859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671404 4859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671407 4859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671409 4859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671412 4859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671415 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671417 4859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671420 4859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671422 4859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671425 4859 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671427 4859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671430 4859 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671432 4859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671435 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671437 4859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671440 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671442 4859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:22.685325 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671447 4859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671451 4859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671455 4859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671458 4859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671460 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671463 4859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671465 4859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671468 4859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671470 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671473 4859 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671476 4859 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671478 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671481 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671483 4859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671487 4859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671490 4859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671494 4859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671496 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671499 4859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671502 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:22.685963 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671504 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671507 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671509 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671512 4859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671514 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671517 4859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671520 4859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671522 4859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671525 4859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671527 4859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671530 4859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671533 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671535 4859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671538 4859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671541 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:22.686473 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.671564 4859 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671598 4859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671603 4859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671606 4859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671611 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671614 4859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671617 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671620 4859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671622 4859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671625 4859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671628 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671631 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671636 4859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671640 4859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671644 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671648 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671652 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671656 4859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671660 4859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:22.687076 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671663 4859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671668 4859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671671 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671673 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671676 4859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671678 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671681 4859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671684 4859 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671686 4859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671689 4859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671691 4859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671694 4859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671697 4859 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671700 4859 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671703 4859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671705 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671708 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671712 4859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671714 4859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671717 4859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:22.687628 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671720 4859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671722 4859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671725 4859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671727 4859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671730 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671732 4859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671735 4859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671737 4859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671740 4859 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671743 4859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671745 4859 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671748 4859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671750 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671754 4859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671757 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671759 4859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671763 4859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671767 4859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671770 4859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671772 4859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:22.688275 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671775 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671777 4859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671780 4859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671782 4859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671785 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671789 4859 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671792 4859 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671794 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671796 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671803 4859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671807 4859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671811 4859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671814 4859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671816 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671819 4859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671822 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671825 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671827 4859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671830 4859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:22.688772 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671832 4859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671835 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671838 4859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671840 4859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671843 4859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671845 4859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671850 4859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671852 4859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671855 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671857 4859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671860 4859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671863 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671866 4859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671868 4859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671871 4859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671873 4859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671876 4859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671878 4859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671881 4859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:22.689296 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671884 4859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671886 4859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671889 4859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671893 4859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671897 4859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671900 4859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671902 4859 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671905 4859 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671908 4859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:22.671911 4859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.671981 4859 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.673037 4859 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.673096 4859 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.673282 4859 server.go:1014] "Starting client certificate rotation" Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.673424 4859 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.673465 4859 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:42 +0000 UTC" deadline="2028-07-28 13:25:32.832552324 +0000 UTC" Oct 30 21:42:22.689764 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.673492 4859 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15279h43m10.159062692s" Oct 30 21:42:22.690176 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.673763 4859 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:42:22.690176 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.674867 4859 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:42:22.690176 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.676276 4859 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:42:22.690176 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.677939 4859 log.go:25] "Validated CRI v1 image API" Oct 30 21:42:22.690176 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.678677 4859 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:42:22.690176 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.680512 4859 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 da621e9c-a212-44de-8106-3bd3ab5cb51f:/dev/nvme0n1p3 dbe0243e-a3dc-4862-817c-ff7c58b21379:/dev/nvme0n1p4] Oct 30 21:42:22.690344 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.680534 4859 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04ac54ea25c108341d4c55f6c2444f650cfc4531266e7ef94337cfd4d29c503e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04ac54ea25c108341d4c55f6c2444f650cfc4531266e7ef94337cfd4d29c503e/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2066f57b222fe1f6eb5d43dfcc127adf368a1423896d6f5bc4e77b24d53f18a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2066f57b222fe1f6eb5d43dfcc127adf368a1423896d6f5bc4e77b24d53f18a8/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/243a34521891edabbd7609abf6d5ece7bb4ed4494f514564827a4ac45e8b0c90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/243a34521891edabbd7609abf6d5ece7bb4ed4494f514564827a4ac45e8b0c90/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/246340e635e4734753d5659a7e8d304bbea3282b6444ff08444b7619e28e0958/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/246340e635e4734753d5659a7e8d304bbea3282b6444ff08444b7619e28e0958/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24f4015c9e24b719f689ed33f23316bcb2f8fa795e733e8f856206bd1796c34c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24f4015c9e24b719f689ed33f23316bcb2f8fa795e733e8f856206bd1796c34c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2698d38b98c46a6a617df6ad41dc10fc07f6425a4a63526e494ca8b94ea24409/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2698d38b98c46a6a617df6ad41dc10fc07f6425a4a63526e494ca8b94ea24409/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2feb157933a68a87a88111f706be7e01313449d1e3b7f6d6f3792a8a10e740a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2feb157933a68a87a88111f706be7e01313449d1e3b7f6d6f3792a8a10e740a1/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/318e373e1869f1109dc6c47dcbbedf850d454dd8f4fd0c12ac9d92102bd16b4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/318e373e1869f1109dc6c47dcbbedf850d454dd8f4fd0c12ac9d92102bd16b4e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4402b9278f43b03f38aa205cfe2c5e907880a091a48e006854a0a73b269d0a00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4402b9278f43b03f38aa205cfe2c5e907880a091a48e006854a0a73b269d0a00/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/553b89959a6df1f69f57893dd4ef378776ca7d9d88258ef9e4dc33d9de386559/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/553b89959a6df1f69f57893dd4ef378776ca7d9d88258ef9e4dc33d9de386559/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60fbc58e6170538cfe71c93892d3bcc81b0c1d70980153533671ac93b0325ad9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60fbc58e6170538cfe71c93892d3bcc81b0c1d70980153533671ac93b0325ad9/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b7c5d8b289c438aa7c0887e8172787cd7244bbd78a841f57b10cae4bed7427a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b7c5d8b289c438aa7c0887e8172787cd7244bbd78a841f57b10cae4bed7427a/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c21b8f7a1bdfbae68dc916f3a42a5045081f3dc6a67915dc59d01361ea7c12a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c21b8f7a1bdfbae68dc916f3a42a5045081f3dc6a67915dc59d01361ea7c12a/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79290572886bebe8b8d943a25b6c65463c351392ee2855ef3aa21f15f9ba35ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79290572886bebe8b8d943a25b6c65463c351392ee2855ef3aa21f15f9ba35ca/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79b26fad3b2a36dc9b8d8aef472ed8513da4cba7a5c37647a7f90832a85d147a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79b26fad3b2a36dc9b8d8aef472ed8513da4cba7a5c37647a7f90832a85d147a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80cca2c6f465b9aee7c53bca624627e74f650c28693b6b70f26804479f89109e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80cca2c6f465b9aee7c53bca624627e74f650c28693b6b70f26804479f89109e/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b34d7d0d5ccdeacfd4672d95203f4194409a25f2597a325575c02d3ce076fb9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b34d7d0d5ccdeacfd4672d95203f4194409a25f2597a325575c02d3ce076fb9a/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8377cf2a5da96a735d8ed758cc5bf91d3d741598a266d8073a391bbd14170da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8377cf2a5da96a735d8ed758cc5bf91d3d741598a266d8073a391bbd14170da/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca36c6a2f3eeb0ef49de9429261ceaa55020e345526187e5916069c2348176a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca36c6a2f3eeb0ef49de9429261ceaa55020e345526187e5916069c2348176a4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d277dc06d694c1abed779199fbc98886281c01de739b0676bb48321205256450/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d277dc06d694c1abed779199fbc98886281c01de739b0676bb48321205256450/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d89e4ed6631fbd05fe2eef9e0c60ef3712d511f053ec7fa47506593fa3e7ace2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d89e4ed6631fbd05fe2eef9e0c60ef3712d511f053ec7fa47506593fa3e7ace2/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dea1b692b04a74ac2f4122ee19064d6d821565a931a3e98b1cdb953c0421fedc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dea1b692b04a74ac2f4122ee19064d6d821565a931a3e98b1cdb953c0421fedc/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12730f3a-4d15-47f1-af22-9f4e37eedac7/volumes/kubernetes.io~projected/kube-api-access-c9cgw:{mountpoint:/var/lib/kubelet/pods/12730f3a-4d15-47f1-af22-9f4e37eedac7/volumes/kubernetes.io~projected/kube-api-access-c9cgw major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a631286-ec63-4b0a-8fc6-f97280e181a1/volumes/kubernetes.io~projected/kube-api-access-7gq6m:{mountpoint:/var/lib/kubelet/pods/1a631286-ec63-4b0a-8fc6-f97280e181a1/volumes/kubernetes.io~projected/kube-api-access-7gq6m major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d29a054-4a50-4709-81c8-01c50df2944e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1d29a054-4a50-4709-81c8-01c50df2944e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d29a054-4a50-4709-81c8-01c50df2944e/volumes/kubernetes.io~projected/kube-api-access-54q8h:{mountpoint:/var/lib/kubelet/pods/1d29a054-4a50-4709-81c8-01c50df2944e/volumes/kubernetes.io~projected/kube-api-access-54q8h major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d29a054-4a50-4709-81c8-01c50df2944e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1d29a054-4a50-4709-81c8-01c50df2944e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22242d39-b0fc-48a0-a9f2-9a28f7106fbd/volumes/kubernetes.io~projected/kube-api-access-tk7q5:{mountpoint:/var/lib/kubelet/pods/22242d39-b0fc-48a0-a9f2-9a28f7106fbd/volumes/kubernetes.io~projected/kube-api-access-tk7q5 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a85ea4e-16f2-4088-ba73-7e996f46e151/volumes/kubernetes.io~projected/kube-api-access-z6p77:{mountpoint:/var/lib/kubelet/pods/3a85ea4e-16f2-4088-ba73-7e996f46e151/volumes/kubernetes.io~projected/kube-api-access-z6p77 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bcc5627-b74e-49ed-a82b-36c2920b9917/volumes/kubernetes.io~projected/kube-api-access-654cq:{mountpoint:/var/lib/kubelet/pods/3bcc5627-b74e-49ed-a82b-36c2920b9917/volumes/kubernetes.io~projected/kube-api-access-654cq major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bcc5627-b74e-49ed-a82b-36c2920b9917/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3bcc5627-b74e-49ed-a82b-36c2920b9917/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f03966a-97af-46f0-99a5-d50a4f54e744/volumes/kubernetes.io~projected/kube-api-access-8mt5k:{mountpoint:/var/lib/kubelet/pods/3f03966a-97af-46f0-99a5-d50a4f54e744/volumes/kubernetes.io~projected/kube-api-access-8mt5k major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40cea1c9-e71c-4451-926c-398f36e9691c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/40cea1c9-e71c-4451-926c-398f36e9691c/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5197a548-5e64-49a3-a5b4-421898abe7df/volumes/kubernetes.io~projected/kube-api-access-m8gcb:{mountpoint:/var/lib/kubelet/pods/5197a548-5e64-49a3-a5b4-421898abe7df/volumes/kubernetes.io~projected/kube-api-access-m8gcb major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200dd70-ca15-4547-b344-c772e014e92b/volumes/kubernetes.io~projected/kube-api-access-q8lr5:{mountpoint:/var/lib/kubelet/pods/5200dd70-ca15-4547-b344-c772e014e92b/volumes/kubernetes.io~projected/kube-api-access-q8lr5 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5387b573-d7bf-47bc-9d20-755835781b5b/volumes/kubernetes.io~projected/kube-api-access-7bnrn:{mountpoint:/var/lib/kubelet/pods/5387b573-d7bf-47bc-9d20-755835781b5b/volumes/kubernetes.io~projected/kube-api-access-7bnrn major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5528ca3c-d24a-485c-b896-54fdc2fdb5ac/volumes/kubernetes.io~projected/kube-api-access-lgn5s:{mountpoint:/var/lib/kubelet/pods/5528ca3c-d24a-485c-b896-54fdc2fdb5ac/volumes/kubernetes.io~projected/kube-api-access-lgn5s major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56e7aa71-e129-4ceb-91a9-06770acc40f7/volumes/kubernetes.io~projected/kube-api-access-8pnln:{mountpoint:/var/lib/kubelet/pods/56e7aa71-e129-4ceb-91a9-06770acc40f7/volumes/kubernetes.io~projected/kube-api-access-8pnln major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56e7aa71-e129-4ceb-91a9-06770acc40f7/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/56e7aa71-e129-4ceb-91a9-06770acc40f7/volumes/kubernetes.io~secret/default-certificate major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56e7aa71-e129-4ceb-91a9-06770acc40f7/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/56e7aa71-e129-4ceb-91a9-06770acc40f7/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61f77d06-9723-4c98-8dd8-80c01d978e2f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/61f77d06-9723-4c98-8dd8-80c01d978e2f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61f77d06-9723-4c98-8dd8-80c01d978e2f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/61f77d06-9723-4c98-8dd8-80c01d978e2f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61f77d06-9723-4c98-8dd8-80c01d978e2f/volumes/kubernetes.io~projected/kube-api-access-zgqcc:{mountpoint:/var/lib/kubelet/pods/61f77d06-9723-4c98-8dd8-80c01d978e2f/volumes/kubernetes.io~projected/kube-api-access-zgqcc major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6307df15-18fe-45cb-b509-298bcb1fcf3b/volumes/kubernetes.io~projected/kube-api-access-tbcw8:{mountpoint:/var/lib/kubelet/pods/6307df15-18fe-45cb-b509-298bcb1fcf3b/volumes/kubernetes.io~projected/kube-api-access-tbcw8 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6307df15-18fe-45cb-b509-298bcb1fcf3b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6307df15-18fe-45cb-b509-298bcb1fcf3b/volumes/kubernetes.io~secret/cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/641871f0-a0d5-4be1-b2d0-3c7603211724/volumes/kubernetes.io~projected/kube-api-access-gqm4k:{mountpoint:/var/lib/kubelet/pods/641871f0-a0d5-4be1-b2d0-3c7603211724/volumes/kubernetes.io~projected/kube-api-access-gqm4k major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c914e8-45f5-4219-9790-b5fc849d39f7/volumes/kubernetes.io~projected/kube-api-access-txqg8:{mountpoint:/var/lib/kubelet/pods/65c914e8-45f5-4219-9790-b5fc849d39f7/volumes/kubernetes.io~projected/kube-api-access-txqg8 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~projected/kube-api-access-t7l4k:{mountpoint:/var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~projected/kube-api-access-t7l4k major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70fbe0ee-70e4-47b1-8094-4faa96292c6b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/70fbe0ee-70e4-47b1-8094-4faa96292c6b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78532fdc-d35e-4ec3-a48a-18fe0bbe8484/volumes/kubernetes.io~projected/kube-api-access-swxd6:{mountpoint:/var/lib/kubelet/pods/78532fdc-d35e-4ec3-a48a-18fe0bbe8484/volumes/kubernetes.io~projected/kube-api-access-swxd6 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0/volumes/kubernetes.io~projected/kube-api-access-dg55q:{mountpoint:/var/lib/kubelet/pods/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0/volumes/kubernetes.io~projected/kube-api-access-dg55q major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81fb1745-4867-438d-b4e0-d44e743e5230/volumes/kubernetes.io~projected/kube-api-access-4rvjt:{mountpoint:/var/lib/kubelet/pods/81fb1745-4867-438d-b4e0-d44e743e5230/volumes/kubernetes.io~projected/kube-api-access-4rvjt major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81fb1745-4867-438d-b4e0-d44e743e5230/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/81fb1745-4867-438d-b4e0-d44e743e5230/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~projected/kube-api-access-58245:{mountpoint:/var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~projected/kube-api-access-58245 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a403f7b4-a616-4807-8263-678045389f84/volumes/kubernetes.io~projected/kube-api-access-gfdbg:{mountpoint:/var/lib/kubelet/pods/a403f7b4-a616-4807-8263-678045389f84/volumes/kubernetes.io~projected/kube-api-access-gfdbg major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a749e90b-1748-48e1-b809-89cece3bab0a/volumes/kubernetes.io~projected/kube-api-access-zfw6m:{mountpoint:/var/lib/kubelet/pods/a749e90b-1748-48e1-b809-89cece3bab0a/volumes/kubernetes.io~projected/kube-api-access-zfw6m major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aff7cebc-f18e-4b88-bb91-bec7ed9ab097/volumes/kubernetes.io~projected/kube-api-access-wqhwd:{mountpoint:/var/lib/kubelet/pods/aff7cebc-f18e-4b88-bb91-bec7ed9ab097/volumes/kubernetes.io~projected/kube-api-access-wqhwd major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21892bd-c791-419a-a1d7-4bdc643b7f07/volumes/kubernetes.io~projected/kube-api-access-kzqbm:{mountpoint:/var/lib/kubelet/pods/b21892bd-c791-419a-a1d7-4bdc643b7f07/volumes/kubernetes.io~projected/kube-api-access-kzqbm major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21892bd-c791-419a-a1d7-4bdc643b7f07/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b21892bd-c791-419a-a1d7-4bdc643b7f07/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24/volumes/kubernetes.io~projected/kube-api-access-45k86:{mountpoint:/var/lib/kubelet/pods/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24/volumes/kubernetes.io~projected/kube-api-access-45k86 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9d15d9a-71ac-4572-98ca-c5f5df251cb2/volumes/kubernetes.io~projected/kube-api-access-j4f9s:{mountpoint:/var/lib/kubelet/pods/f9d15d9a-71ac-4572-98ca-c5f5df251cb2/volumes/kubernetes.io~projected/kube-api-access-j4f9s major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9d15d9a-71ac-4572-98ca-c5f5df251cb2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f9d15d9a-71ac-4572-98ca-c5f5df251cb2/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d8d365170d80d3f0e94ca62bf54b61c846637d3b20a356bc6a5cb4bc1dbf263e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ca8cd529adb6761dae958b6fdb736ce31e6f566c935f0da62411995acbaf38ad/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/bd7b9f69a676f6aa09410efdb5c0d762c627c1afca2e15b2bde699bcc19b6ad2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/ce23b40190e6bf5e4556595051c4aab4121f3d09086c809d45050191f90b480f/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/ffd738ada72553572023d85c45179cdd0531a15c057b257b02029f728b1ec960/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f7734394d290616530ec513c8bba00681b3e9fcd4d79350b1172a8a9f234bd45/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6f70eaf5884356541de4c5abd2a29bdabb79b0cb72641b61098a533bffc0133f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1498fb4b8b7d97ea73fb1c32761ba1f356b1c22f86bfcf5d0e7061c38fdacfb2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8fa19ca8d696b200219699169713e8c8a98d4eb5b48e0cc2ad0cd6cf338c61b1/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c06798aa3128d14c98ddeee0d0cf5eb4b044f7e600efae1d0065df716b51701d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/30bf6dd7c5b0c99f504f41a80b4cfd0c0428f286ae61847babce1afd58ae2d0e/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/cb064ea3813e9b17711a67839c2ebf5f2d4188dc2ffd2c1d35f0b900f2a97261/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/89d865dcb52d001e278214319d07ee5ca523af697b47074a19aba3aa73230934/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7908bf15f2b7d68fa94547a4661f33aaf9592f603b7c290e5014e9891f886bb9/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6ff18c35d5797d348141d582de4f9a43374f5783cb911381009033e99c1f08f7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/15072bfb7667f04507e611fb57dae3e87b07a238d63c19e5d4620182badcc938/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5bbd6f5424a4340b64de1949e16a055577117808c7a2d61d56f58be6db7b80bd/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1c1a96e971034615620e87807a7ebe8af0b2cc7007ea05fba34bb231ae3f1561/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/21246bd54e967e382a9916be5fd25a5a8bd433ffbb60574368150d82ae5ce741/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/65fa59ee9140440a1aa782110e3a4ee969caba254a5f98d8d2cc6b9fa0ea6987/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/87cd7c7ba44b182a28e1f7512d70bd6349d9fbf51ed5bfbd3d5bdbaeb79bbd10/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/45321a6ea2a30cfa215181e60d27bf69a621a7c7d4d45460bddb001071f19e61/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/a20df616ad2fe41e31b8014101abe0b073a9cfc31f09342f9dc8f6f9b355dac3/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/cfc3e6305ab929d30d19bf8e54cf38eb54a7fc34b96dbd11094ce0de5e0550ff/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/1d74ee4711c145ad506d4ba9d4b1a4d1fb575b2a884016e9d70129adba478ca2/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/17bffefcc432273bcd9595ca76e5b4a6d3ad9028bbab467ac6668a5c53d04e9c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d899fd8bea19fac6e647677d795f1e2bbae5acf57519a599fbe7d45b7c69e97d/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/632387a175a0ef93c374334123970ee12bf43b4cd4224f0dd24ab85d1648852e/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/a6faea8bdbabe777e1bd03f651d834b586259f4d06c4bf2e5cc7537563cf56d8/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/55057c9abf0e9b546662bbaa59d5c153ca4e2e432003467592021061dfd15942/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/c8b35ccb1ddb4849df9d7568091ddfe12a106498a30dc35a768e7a88f8b480c5/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/d340d4b75d105d376364c731d1753d9ed79e6e1b0c49a09be8ca832371182fb9/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/f8a8a5137a8fe4c9440ebcc9476efd2669986e6b617c8a7c84e7cec24164afc1/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/8f0d53f865a8010f589af593fdef1308c818f62bb377102262ab0db8a9120555/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/c0b2d91666365846846b86460de9791e057bb8851d14b5cde6052a94788ec15a/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/e8844ce438e73e47c6bea4fe66f8506294145778d2128c83833131ffb39fb101/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/d14d08b18ae5aa3574d9aa88be0ec828103ef76049566010182527b82ddf38a7/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/7fb492fae83316731b6d0c16424706813ad69e30ce84c8464b7a02af4b65625d/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/299d3cd5c125848b490e115c74e5d25ad6201b512f6e3e47c489083a6e0fbb77/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/4119062e7f6267294bb57f9605ce34f7bf67acb4cf22fa122bfd0ae5d5318d10/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/aaa9d736d2d7c2cd18881a582e21993abc6753892f1457791d9f6f41038227fe/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/53a8d60b85cce1401bb300773bb8b6a83e1d2742a2653c1502b26ddd7f195a57/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/6da2a9d4c4c7c15f6d9d397f9b5675dcc0c704ef7fe52ab44b9a72018789f82b/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/ace707b27f51cc51fe2a7ba16b38124658583c1d28681fac13cedce8ccc11019/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/c03a6d6a66cb9b4ed2153041746b5960d26c158e17483251196e18abec497789/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/c92a554bea589b7505d8e1e12f59ccf86197e2a88562df267eef550d603c5fc9/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0ee21895a26d0a39e10d8dab995796cb73f28def843022fccaeba7ff6e851255/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ee0610aebe12e80997cd74cc31f36b4d3c4ad05a4034ba51b1f4fe634d805460/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/57012ae0361ddaaf2e1e344481323e6a34919d58ba280dfd4f73a58023424763/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ae28b3d27779961bc4746aa0f56d49111b59bc16397fa52e43df760dc070f0c0/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9d75be5f28366d96e645d0e331ac8f6bd2a3b099c9963e61cad95c7701eac2b9/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/509381144bcfca6a9dd8fa9d131c13786448ee34901eb760b86268a89eb92cba/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/2febb28f1abdbbce180f7280b3c86813014b8423b604ed14288d607c22cd7b86/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b0c956ac00ac079b43cb0a2eb9a6a1daf957b0b9a8712b39078a794acbe2a2f4/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:42:22.690577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.690145 4859 manager.go:217] Machine: {Timestamp:2026-10-30 21:42:22.68965352 +0000 UTC m=+0.062280261 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099856 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec283ca0bea5853aeeef5080fba298b2 SystemUUID:ec283ca0-bea5-853a-eeef-5080fba298b2 BootID:404ce9a8-348c-423c-9019-6dec705796a8 Filesystems:[{Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/81fb1745-4867-438d-b4e0-d44e743e5230/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5528ca3c-d24a-485c-b896-54fdc2fdb5ac/volumes/kubernetes.io~projected/kube-api-access-lgn5s DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80cca2c6f465b9aee7c53bca624627e74f650c28693b6b70f26804479f89109e/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24f4015c9e24b719f689ed33f23316bcb2f8fa795e733e8f856206bd1796c34c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81fb1745-4867-438d-b4e0-d44e743e5230/volumes/kubernetes.io~projected/kube-api-access-4rvjt DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d89e4ed6631fbd05fe2eef9e0c60ef3712d511f053ec7fa47506593fa3e7ace2/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/553b89959a6df1f69f57893dd4ef378776ca7d9d88258ef9e4dc33d9de386559/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79b26fad3b2a36dc9b8d8aef472ed8513da4cba7a5c37647a7f90832a85d147a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22242d39-b0fc-48a0-a9f2-9a28f7106fbd/volumes/kubernetes.io~projected/kube-api-access-tk7q5 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dea1b692b04a74ac2f4122ee19064d6d821565a931a3e98b1cdb953c0421fedc/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6307df15-18fe-45cb-b509-298bcb1fcf3b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bcc5627-b74e-49ed-a82b-36c2920b9917/volumes/kubernetes.io~projected/kube-api-access-654cq DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2066f57b222fe1f6eb5d43dfcc127adf368a1423896d6f5bc4e77b24d53f18a8/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9d15d9a-71ac-4572-98ca-c5f5df251cb2/volumes/kubernetes.io~projected/kube-api-access-j4f9s DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b21892bd-c791-419a-a1d7-4bdc643b7f07/volumes/kubernetes.io~projected/kube-api-access-kzqbm DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40cea1c9-e71c-4451-926c-398f36e9691c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a631286-ec63-4b0a-8fc6-f97280e181a1/volumes/kubernetes.io~projected/kube-api-access-7gq6m DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5200dd70-ca15-4547-b344-c772e014e92b/volumes/kubernetes.io~projected/kube-api-access-q8lr5 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79290572886bebe8b8d943a25b6c65463c351392ee2855ef3aa21f15f9ba35ca/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a749e90b-1748-48e1-b809-89cece3bab0a/volumes/kubernetes.io~projected/kube-api-access-zfw6m DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8377cf2a5da96a735d8ed758cc5bf91d3d741598a266d8073a391bbd14170da/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56e7aa71-e129-4ceb-91a9-06770acc40f7/volumes/kubernetes.io~projected/kube-api-access-8pnln DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a403f7b4-a616-4807-8263-678045389f84/volumes/kubernetes.io~projected/kube-api-access-gfdbg DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b7c5d8b289c438aa7c0887e8172787cd7244bbd78a841f57b10cae4bed7427a/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24/volumes/kubernetes.io~projected/kube-api-access-45k86 DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/61f77d06-9723-4c98-8dd8-80c01d978e2f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d277dc06d694c1abed779199fbc98886281c01de739b0676bb48321205256450/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0/volumes/kubernetes.io~projected/kube-api-access-dg55q DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6307df15-18fe-45cb-b509-298bcb1fcf3b/volumes/kubernetes.io~projected/kube-api-access-tbcw8 DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61f77d06-9723-4c98-8dd8-80c01d978e2f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a85ea4e-16f2-4088-ba73-7e996f46e151/volumes/kubernetes.io~projected/kube-api-access-z6p77 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b21892bd-c791-419a-a1d7-4bdc643b7f07/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d29a054-4a50-4709-81c8-01c50df2944e/volumes/kubernetes.io~projected/kube-api-access-54q8h DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d29a054-4a50-4709-81c8-01c50df2944e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca36c6a2f3eeb0ef49de9429261ceaa55020e345526187e5916069c2348176a4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/243a34521891edabbd7609abf6d5ece7bb4ed4494f514564827a4ac45e8b0c90/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/61f77d06-9723-4c98-8dd8-80c01d978e2f/volumes/kubernetes.io~projected/kube-api-access-zgqcc DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aff7cebc-f18e-4b88-bb91-bec7ed9ab097/volumes/kubernetes.io~projected/kube-api-access-wqhwd DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78532fdc-d35e-4ec3-a48a-18fe0bbe8484/volumes/kubernetes.io~projected/kube-api-access-swxd6 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56e7aa71-e129-4ceb-91a9-06770acc40f7/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3f03966a-97af-46f0-99a5-d50a4f54e744/volumes/kubernetes.io~projected/kube-api-access-8mt5k DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70fbe0ee-70e4-47b1-8094-4faa96292c6b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:266 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56e7aa71-e129-4ceb-91a9-06770acc40f7/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9d15d9a-71ac-4572-98ca-c5f5df251cb2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65c914e8-45f5-4219-9790-b5fc849d39f7/volumes/kubernetes.io~projected/kube-api-access-txqg8 DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b34d7d0d5ccdeacfd4672d95203f4194409a25f2597a325575c02d3ce076fb9a/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2feb157933a68a87a88111f706be7e01313449d1e3b7f6d6f3792a8a10e740a1/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2698d38b98c46a6a617df6ad41dc10fc07f6425a4a63526e494ca8b94ea24409/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c21b8f7a1bdfbae68dc916f3a42a5045081f3dc6a67915dc59d01361ea7c12a/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/318e373e1869f1109dc6c47dcbbedf850d454dd8f4fd0c12ac9d92102bd16b4e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/246340e635e4734753d5659a7e8d304bbea3282b6444ff08444b7619e28e0958/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes/kubernetes.io~projected/kube-api-access-t7l4k DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60fbc58e6170538cfe71c93892d3bcc81b0c1d70980153533671ac93b0325ad9/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04ac54ea25c108341d4c55f6c2444f650cfc4531266e7ef94337cfd4d29c503e/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/641871f0-a0d5-4be1-b2d0-3c7603211724/volumes/kubernetes.io~projected/kube-api-access-gqm4k DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4402b9278f43b03f38aa205cfe2c5e907880a091a48e006854a0a73b269d0a00/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5387b573-d7bf-47bc-9d20-755835781b5b/volumes/kubernetes.io~projected/kube-api-access-7bnrn DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12730f3a-4d15-47f1-af22-9f4e37eedac7/volumes/kubernetes.io~projected/kube-api-access-c9cgw DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes/kubernetes.io~projected/kube-api-access-58245 DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d29a054-4a50-4709-81c8-01c50df2944e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3bcc5627-b74e-49ed-a82b-36c2920b9917/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5197a548-5e64-49a3-a5b4-421898abe7df/volumes/kubernetes.io~projected/kube-api-access-m8gcb DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:04ac54ea25c1083 MacAddress:4a:67:b6:51:ec:29 Speed:10000 Mtu:8901} {Name:2066f57b222fe1f MacAddress:52:cc:88:c2:bd:7a Speed:10000 Mtu:8901} {Name:318e373e1869f11 MacAddress:ae:72:cf:0b:ee:af Speed:10000 Mtu:8901} {Name:60fbc58e6170538 MacAddress:fa:77:52:db:94:ac Speed:10000 Mtu:8901} {Name:6b7c5d8b289c438 MacAddress:c6:eb:cd:75:ec:37 Speed:10000 Mtu:8901} {Name:6c21b8f7a1bdfba MacAddress:9e:02:1d:d0:be:2a Speed:10000 Mtu:8901} {Name:80cca2c6f465b9a MacAddress:06:96:70:ad:15:5e Speed:10000 Mtu:8901} {Name:b34d7d0d5ccdeac MacAddress:7a:84:c0:29:98:77 Speed:10000 Mtu:8901} {Name:b8377cf2a5da96a MacAddress:fa:b5:28:b9:7e:b7 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:19:4e:ba:b0:53 Speed:0 Mtu:9001} {Name:br-int MacAddress:62:ea:ea:a2:f2:03 Speed:0 Mtu:8901} {Name:d277dc06d694c1a MacAddress:f6:d4:fd:e9:ba:65 Speed:10000 Mtu:8901} {Name:d89e4ed6631fbd0 MacAddress:76:3e:ad:56:d7:06 Speed:10000 Mtu:8901} {Name:dea1b692b04a74a MacAddress:76:11:db:3f:33:44 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:19:4e:ba:b0:53 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:f6:f4:59:3e:bb:5a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:49:c3:bf:88:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:42:22.690577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.690499 4859 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:42:22.690577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.690548 4859 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:42:22.690904 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.690876 4859 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:42:22.691048 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.690904 4859 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-104.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:42:22.691103 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691050 4859 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:42:22.691103 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691058 4859 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:42:22.691103 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691070 4859 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:42:22.691103 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691090 4859 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:42:22.691210 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691186 4859 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:42:22.691389 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691378 4859 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:42:22.691507 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691498 4859 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:42:22.691543 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691535 4859 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:42:22.691571 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691548 4859 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:42:22.691571 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691558 4859 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:42:22.691571 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691567 4859 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:42:22.691660 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.691575 4859 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:42:22.692435 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.692415 4859 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:42:22.692435 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.692433 4859 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:42:22.692985 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.692967 4859 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:42:22.695444 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695431 4859 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:42:22.695444 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695444 4859 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:42:22.695547 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695537 4859 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:42:22.695750 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695739 4859 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:42:22.695783 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695768 4859 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:42:22.695866 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695857 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:42:22.695897 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695872 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:42:22.695897 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695878 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:42:22.695961 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695906 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:42:22.695961 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695915 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:42:22.695961 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695924 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:42:22.695961 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695932 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:42:22.695961 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695938 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:42:22.695961 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695961 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:42:22.696141 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695968 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:42:22.696141 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695984 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:42:22.696141 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.695994 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:42:22.696141 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.696020 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:42:22.696141 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.696028 4859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:42:22.699749 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.699734 4859 server.go:1263] "Started kubelet" Oct 30 21:42:22.700229 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.699836 4859 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:42:22.700366 ip-10-0-141-104 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:42:22.700699 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.700403 4859 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:42:22.700699 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.700498 4859 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:42:22.700878 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.700845 4859 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:42:22.704777 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.704047 4859 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:42:22.709521 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.709042 4859 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:42:22.709521 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.709065 4859 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:42:22.709521 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.709113 4859 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:43 +0000 UTC" deadline="2028-08-02 00:56:24.250401678 +0000 UTC" Oct 30 21:42:22.709521 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.709133 4859 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15387h14m1.541273721s" Oct 30 21:42:22.709521 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.709157 4859 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:42:22.709521 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.709168 4859 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:42:22.709521 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:22.709197 4859 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-104.us-west-1.compute.internal\" not found" Oct 30 21:42:22.709521 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.709267 4859 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:42:22.710047 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.709934 4859 factory.go:55] Registering systemd factory Oct 30 21:42:22.710047 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.710005 4859 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.711069 4859 factory.go:153] Registering CRI-O factory Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.711089 4859 factory.go:223] Registration of the crio container factory successfully Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.711167 4859 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.711191 4859 factory.go:103] Registering Raw factory Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.711209 4859 manager.go:1199] Started watching for new ooms in manager Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.711553 4859 manager.go:319] Starting recovery of all containers Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712164 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78532fdc-d35e-4ec3-a48a-18fe0bbe8484" volumeName="kubernetes.io/projected/78532fdc-d35e-4ec3-a48a-18fe0bbe8484-kube-api-access-swxd6" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712201 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a474868-17e7-4ad9-a3f7-65376fc52940" volumeName="kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712218 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a403f7b4-a616-4807-8263-678045389f84" volumeName="kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712233 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" volumeName="kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-trusted-ca" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712245 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a631286-ec63-4b0a-8fc6-f97280e181a1" volumeName="kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-cni-binary-copy" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712258 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d29a054-4a50-4709-81c8-01c50df2944e" volumeName="kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-env-overrides" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712270 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200dd70-ca15-4547-b344-c772e014e92b" volumeName="kubernetes.io/projected/5200dd70-ca15-4547-b344-c772e014e92b-kube-api-access-q8lr5" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712282 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c914e8-45f5-4219-9790-b5fc849d39f7" volumeName="kubernetes.io/projected/65c914e8-45f5-4219-9790-b5fc849d39f7-kube-api-access-txqg8" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712297 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" volumeName="kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712310 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81fb1745-4867-438d-b4e0-d44e743e5230" volumeName="kubernetes.io/configmap/81fb1745-4867-438d-b4e0-d44e743e5230-config" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712323 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d29a054-4a50-4709-81c8-01c50df2944e" volumeName="kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712336 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d29a054-4a50-4709-81c8-01c50df2944e" volumeName="kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-config" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712349 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d29a054-4a50-4709-81c8-01c50df2944e" volumeName="kubernetes.io/secret/1d29a054-4a50-4709-81c8-01c50df2944e-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712363 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="641871f0-a0d5-4be1-b2d0-3c7603211724" volumeName="kubernetes.io/projected/641871f0-a0d5-4be1-b2d0-3c7603211724-kube-api-access-gqm4k" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712382 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21892bd-c791-419a-a1d7-4bdc643b7f07" volumeName="kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-service-ca-bundle" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712393 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21892bd-c791-419a-a1d7-4bdc643b7f07" volumeName="kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712405 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21892bd-c791-419a-a1d7-4bdc643b7f07" volumeName="kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-tmp" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712418 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12730f3a-4d15-47f1-af22-9f4e37eedac7" volumeName="kubernetes.io/configmap/12730f3a-4d15-47f1-af22-9f4e37eedac7-config-volume" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712429 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a631286-ec63-4b0a-8fc6-f97280e181a1" volumeName="kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-daemon-config" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712441 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5528ca3c-d24a-485c-b896-54fdc2fdb5ac" volumeName="kubernetes.io/projected/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kube-api-access-lgn5s" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712453 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a474868-17e7-4ad9-a3f7-65376fc52940" volumeName="kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-certificates" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712466 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d29a054-4a50-4709-81c8-01c50df2944e" volumeName="kubernetes.io/projected/1d29a054-4a50-4709-81c8-01c50df2944e-kube-api-access-54q8h" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712479 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70fbe0ee-70e4-47b1-8094-4faa96292c6b" volumeName="kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712493 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a474868-17e7-4ad9-a3f7-65376fc52940" volumeName="kubernetes.io/empty-dir/8a474868-17e7-4ad9-a3f7-65376fc52940-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712507 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a474868-17e7-4ad9-a3f7-65376fc52940" volumeName="kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-kube-api-access-58245" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712519 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a749e90b-1748-48e1-b809-89cece3bab0a" volumeName="kubernetes.io/projected/a749e90b-1748-48e1-b809-89cece3bab0a-kube-api-access-zfw6m" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712540 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff7cebc-f18e-4b88-bb91-bec7ed9ab097" volumeName="kubernetes.io/projected/aff7cebc-f18e-4b88-bb91-bec7ed9ab097-kube-api-access-wqhwd" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712555 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21892bd-c791-419a-a1d7-4bdc643b7f07" volumeName="kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-snapshots" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712582 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12730f3a-4d15-47f1-af22-9f4e37eedac7" volumeName="kubernetes.io/empty-dir/12730f3a-4d15-47f1-af22-9f4e37eedac7-tmp-dir" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712654 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f03966a-97af-46f0-99a5-d50a4f54e744" volumeName="kubernetes.io/projected/3f03966a-97af-46f0-99a5-d50a4f54e744-kube-api-access-8mt5k" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712671 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61f77d06-9723-4c98-8dd8-80c01d978e2f" volumeName="kubernetes.io/projected/61f77d06-9723-4c98-8dd8-80c01d978e2f-kube-api-access-zgqcc" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712682 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699c93ce-4424-4460-8cb2-b5db6ca24033" volumeName="kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-kube-api-access-t7l4k" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712710 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a" volumeName="kubernetes.io/configmap/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-nginx-conf" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712725 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2c6a540-c40a-4a1e-b7c6-70e4384f1f24" volumeName="kubernetes.io/projected/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-api-access-45k86" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712739 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" volumeName="kubernetes.io/projected/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-kube-api-access-j4f9s" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712762 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6307df15-18fe-45cb-b509-298bcb1fcf3b" volumeName="kubernetes.io/secret/6307df15-18fe-45cb-b509-298bcb1fcf3b-cert" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712787 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699c93ce-4424-4460-8cb2-b5db6ca24033" volumeName="kubernetes.io/empty-dir/699c93ce-4424-4460-8cb2-b5db6ca24033-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712811 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" volumeName="kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-binary-copy" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712834 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a474868-17e7-4ad9-a3f7-65376fc52940" volumeName="kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-trusted-ca" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712847 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40cea1c9-e71c-4451-926c-398f36e9691c" volumeName="kubernetes.io/configmap/40cea1c9-e71c-4451-926c-398f36e9691c-konnectivity-ca" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712862 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40cea1c9-e71c-4451-926c-398f36e9691c" volumeName="kubernetes.io/secret/40cea1c9-e71c-4451-926c-398f36e9691c-agent-certs" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712876 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200dd70-ca15-4547-b344-c772e014e92b" volumeName="kubernetes.io/configmap/5200dd70-ca15-4547-b344-c772e014e92b-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712888 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61f77d06-9723-4c98-8dd8-80c01d978e2f" volumeName="kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-tmp" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712899 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699c93ce-4424-4460-8cb2-b5db6ca24033" volumeName="kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712911 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a631286-ec63-4b0a-8fc6-f97280e181a1" volumeName="kubernetes.io/projected/1a631286-ec63-4b0a-8fc6-f97280e181a1-kube-api-access-7gq6m" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712926 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a85ea4e-16f2-4088-ba73-7e996f46e151" volumeName="kubernetes.io/projected/3a85ea4e-16f2-4088-ba73-7e996f46e151-kube-api-access-z6p77" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712970 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f03966a-97af-46f0-99a5-d50a4f54e744" volumeName="kubernetes.io/configmap/3f03966a-97af-46f0-99a5-d50a4f54e744-telemetry-config" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.712999 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56e7aa71-e129-4ceb-91a9-06770acc40f7" volumeName="kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-default-certificate" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713013 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699c93ce-4424-4460-8cb2-b5db6ca24033" volumeName="kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-bound-sa-token" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713026 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2c6a540-c40a-4a1e-b7c6-70e4384f1f24" volumeName="kubernetes.io/empty-dir/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-data-volume" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713070 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" volumeName="kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-config" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713086 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="641871f0-a0d5-4be1-b2d0-3c7603211724" volumeName="kubernetes.io/empty-dir/641871f0-a0d5-4be1-b2d0-3c7603211724-tmp-dir" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713098 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c914e8-45f5-4219-9790-b5fc849d39f7" volumeName="kubernetes.io/configmap/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-config" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713131 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a474868-17e7-4ad9-a3f7-65376fc52940" volumeName="kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-bound-sa-token" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713145 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a85ea4e-16f2-4088-ba73-7e996f46e151" volumeName="kubernetes.io/configmap/3a85ea4e-16f2-4088-ba73-7e996f46e151-serviceca" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713159 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5387b573-d7bf-47bc-9d20-755835781b5b" volumeName="kubernetes.io/projected/5387b573-d7bf-47bc-9d20-755835781b5b-kube-api-access-7bnrn" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713209 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56e7aa71-e129-4ceb-91a9-06770acc40f7" volumeName="kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-stats-auth" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713222 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a474868-17e7-4ad9-a3f7-65376fc52940" volumeName="kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713235 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcc5627-b74e-49ed-a82b-36c2920b9917" volumeName="kubernetes.io/projected/3bcc5627-b74e-49ed-a82b-36c2920b9917-kube-api-access-654cq" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713257 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21892bd-c791-419a-a1d7-4bdc643b7f07" volumeName="kubernetes.io/projected/b21892bd-c791-419a-a1d7-4bdc643b7f07-kube-api-access-kzqbm" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713270 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcc5627-b74e-49ed-a82b-36c2920b9917" volumeName="kubernetes.io/configmap/3bcc5627-b74e-49ed-a82b-36c2920b9917-config" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713286 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61f77d06-9723-4c98-8dd8-80c01d978e2f" volumeName="kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-tuned" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713331 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699c93ce-4424-4460-8cb2-b5db6ca24033" volumeName="kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-trusted-ca" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713345 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" volumeName="kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713358 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12730f3a-4d15-47f1-af22-9f4e37eedac7" volumeName="kubernetes.io/projected/12730f3a-4d15-47f1-af22-9f4e37eedac7-kube-api-access-c9cgw" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713380 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56e7aa71-e129-4ceb-91a9-06770acc40f7" volumeName="kubernetes.io/projected/56e7aa71-e129-4ceb-91a9-06770acc40f7-kube-api-access-8pnln" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713392 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6307df15-18fe-45cb-b509-298bcb1fcf3b" volumeName="kubernetes.io/projected/6307df15-18fe-45cb-b509-298bcb1fcf3b-kube-api-access-tbcw8" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713407 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" volumeName="kubernetes.io/projected/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-kube-api-access-dg55q" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713452 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81fb1745-4867-438d-b4e0-d44e743e5230" volumeName="kubernetes.io/secret/81fb1745-4867-438d-b4e0-d44e743e5230-serving-cert" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713510 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22242d39-b0fc-48a0-a9f2-9a28f7106fbd" volumeName="kubernetes.io/projected/22242d39-b0fc-48a0-a9f2-9a28f7106fbd-kube-api-access-tk7q5" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713541 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcc5627-b74e-49ed-a82b-36c2920b9917" volumeName="kubernetes.io/secret/3bcc5627-b74e-49ed-a82b-36c2920b9917-serving-cert" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713562 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81fb1745-4867-438d-b4e0-d44e743e5230" volumeName="kubernetes.io/projected/81fb1745-4867-438d-b4e0-d44e743e5230-kube-api-access-4rvjt" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713573 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" volumeName="kubernetes.io/secret/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-serving-cert" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713587 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699c93ce-4424-4460-8cb2-b5db6ca24033" volumeName="kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-certificates" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713610 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699c93ce-4424-4460-8cb2-b5db6ca24033" volumeName="kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713627 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21892bd-c791-419a-a1d7-4bdc643b7f07" volumeName="kubernetes.io/secret/b21892bd-c791-419a-a1d7-4bdc643b7f07-serving-cert" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713640 4859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5197a548-5e64-49a3-a5b4-421898abe7df" volumeName="kubernetes.io/projected/5197a548-5e64-49a3-a5b4-421898abe7df-kube-api-access-m8gcb" seLinuxMountContext="" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713658 4859 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:42:22.715544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.713667 4859 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:42:22.731527 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.731481 4859 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:42:22.732989 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.732917 4859 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:42:22.733086 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.733023 4859 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:42:22.733086 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.733047 4859 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:42:22.733143 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:22.733091 4859 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:42:22.737271 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.737207 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/1.log" Oct 30 21:42:22.737498 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.737477 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/1.log" Oct 30 21:42:22.754519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.754479 4859 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="d8e933ad56e4eb3e7ccbf76221ad2eebfdf37847e0e15f82fa13fa6a2f513e28" exitCode=0 Oct 30 21:42:22.754519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.754512 4859 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="c5a8a1aeea37c25b03509f2da19ab118499767d54436e29b46f96e68ded1b815" exitCode=0 Oct 30 21:42:22.754519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.754519 4859 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="271a66aba1ffdf066da8f5365b53d8bb4f8e0c96240e05e6671d0a18972ba145" exitCode=0 Oct 30 21:42:22.754519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.754525 4859 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="8c1c8b32b85fa50bbbd21004b1aecceddbefb286686403b812a86a98a3cebc8e" exitCode=0 Oct 30 21:42:22.754813 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.754531 4859 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="d3e76d9b314584050fc902be93d7a55365595269ff9e7f3a6f062a79d8bc8270" exitCode=0 Oct 30 21:42:22.754813 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.754539 4859 generic.go:356] "Generic (PLEG): container finished" podID="7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0" containerID="f63dbf97bf7747c792f8f516e2c326d0d58f263bb77113d6102756b406c08527" exitCode=0 Oct 30 21:42:22.763705 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.763671 4859 generic.go:356] "Generic (PLEG): container finished" podID="bfcbf966d484c9290a6f563fe77e5e16" containerID="dde92ce8da66cd44cb1c4981d23d0887f7704ef8e0ae5c7288983e44bb35787f" exitCode=0 Oct 30 21:42:22.776384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.776362 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/1.log" Oct 30 21:42:22.776501 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.776391 4859 generic.go:356] "Generic (PLEG): container finished" podID="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" containerID="328a8eaab798437384f355cbcbec523ecb9f13793741b3dada3d785415388da3" exitCode=255 Oct 30 21:42:22.800239 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.800210 4859 manager.go:324] Recovery completed Oct 30 21:42:22.809792 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:22.809733 4859 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-104.us-west-1.compute.internal\" not found" Oct 30 21:42:22.817152 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.817137 4859 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:42:22.818815 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.818796 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:42:22.818899 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.818826 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:42:22.818899 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.818837 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:42:22.823246 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823226 4859 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:42:22.823246 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823240 4859 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:42:22.823384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823249 4859 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:42:22.823384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823258 4859 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:42:22.823384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823376 4859 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:42:22.823502 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823385 4859 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:42:22.823502 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823401 4859 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:42:22.823502 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823406 4859 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:42:22.823502 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823412 4859 policy_none.go:50] "Start" Oct 30 21:42:22.823502 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823420 4859 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:42:22.823502 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823429 4859 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:42:22.823680 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823595 4859 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:42:22.823680 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823603 4859 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:42:22.823680 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.823608 4859 policy_none.go:44] "Start" Oct 30 21:42:22.824178 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.824156 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:42:22.824258 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.824188 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:42:22.824258 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.824199 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:42:22.827828 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.827803 4859 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:42:22.830318 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.830306 4859 manager.go:341] "Starting Device Plugin manager" Oct 30 21:42:22.830372 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:22.830332 4859 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:42:22.830372 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.830341 4859 server.go:87] "Starting device plugin registration server" Oct 30 21:42:22.830661 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.830641 4859 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:42:22.830757 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.830657 4859 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:42:22.830757 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.830738 4859 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:42:22.830919 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.830859 4859 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:42:22.831013 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.830937 4859 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:42:22.831013 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.830972 4859 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:42:22.831200 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.831079 4859 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:42:22.830915616Z","UUID":"70219832-dc20-4dd6-8abb-531cb5fa810b","Handler":null,"Name":"","Endpoint":""} Oct 30 21:42:22.832461 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:22.832436 4859 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:42:22.832560 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.832473 4859 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:42:22.832560 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:22.832482 4859 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-104.us-west-1.compute.internal\" not found" Oct 30 21:42:22.832560 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.832491 4859 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:42:22.833187 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.833149 4859 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal"] Oct 30 21:42:22.833326 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.833296 4859 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:42:22.834227 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.834208 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:42:22.834302 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.834237 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:42:22.834302 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.834254 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:42:22.834603 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.834589 4859 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:42:22.834774 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.834744 4859 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:42:22.835513 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835393 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:42:22.835513 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835415 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:42:22.835513 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835426 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:42:22.835513 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835440 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:42:22.835513 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835459 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:42:22.835513 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835473 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:42:22.835936 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:22.835546 4859 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-104.us-west-1.compute.internal\" not found" node="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:22.835936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835776 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" event={"ID":"bfcbf966d484c9290a6f563fe77e5e16","Type":"ContainerStarted","Data":"8fb4a10cf955fb34ba19be003d9983e883b4dd2a4e93a71e81e42ecb875d0801"} Oct 30 21:42:22.835936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835807 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" event={"ID":"bfcbf966d484c9290a6f563fe77e5e16","Type":"ContainerDied","Data":"dde92ce8da66cd44cb1c4981d23d0887f7704ef8e0ae5c7288983e44bb35787f"} Oct 30 21:42:22.835936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835820 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" event={"ID":"bfcbf966d484c9290a6f563fe77e5e16","Type":"ContainerStarted","Data":"2feb157933a68a87a88111f706be7e01313449d1e3b7f6d6f3792a8a10e740a1"} Oct 30 21:42:22.835936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835853 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" event={"ID":"ff30ba81bde53b1c4da7b24cd2c2a4b9","Type":"ContainerStarted","Data":"02992f0cea2db73002f0e7f8f7902ced6eb157b2226f9c542724ca34808b5f4c"} Oct 30 21:42:22.835936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835867 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" event={"ID":"ff30ba81bde53b1c4da7b24cd2c2a4b9","Type":"ContainerStarted","Data":"79290572886bebe8b8d943a25b6c65463c351392ee2855ef3aa21f15f9ba35ca"} Oct 30 21:42:22.835936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.835886 4859 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:42:22.836570 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.836553 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:42:22.836646 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.836584 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:42:22.836646 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.836599 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:42:22.836733 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:22.836717 4859 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-104.us-west-1.compute.internal\" not found" node="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:22.851255 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.851232 4859 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:42:22.901052 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.901019 4859 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:42:22.931491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.931459 4859 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:42:22.932499 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.932484 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:42:22.932611 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.932514 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:42:22.932611 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.932524 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:42:22.932611 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.932567 4859 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:22.949783 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.949752 4859 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:42:22.950717 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.950698 4859 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:42:22.950907 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:22.950891 4859 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:42:23.001552 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.001517 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.001691 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.001576 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.001691 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.001631 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff30ba81bde53b1c4da7b24cd2c2a4b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"ff30ba81bde53b1c4da7b24cd2c2a4b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.012032 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.012006 4859 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.040563 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.040529 4859 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.040713 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.040615 4859 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.041232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.041214 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeNotReady" Oct 30 21:42:23.041317 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.041250 4859 setters.go:546] "Node became not ready" node="ip-10-0-141-104.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:42:23Z","lastTransitionTime":"2026-10-30T21:42:23Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 21:42:23.059926 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.059873 4859 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-104.us-west-1.compute.internal" is owned by different node (oldNodeID="64b6e104-9769-42a3-9b90-fe6d258f580e", newNodeID=""), cleaning up... Oct 30 21:42:23.071754 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.071731 4859 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:42:23.071848 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:23.071782 4859 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.071848 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.071801 4859 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.102484 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.102455 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff30ba81bde53b1c4da7b24cd2c2a4b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"ff30ba81bde53b1c4da7b24cd2c2a4b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.102586 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.102487 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.102586 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.102522 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.102586 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.102566 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.102586 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.102568 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff30ba81bde53b1c4da7b24cd2c2a4b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"ff30ba81bde53b1c4da7b24cd2c2a4b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.102716 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.102568 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfcbf966d484c9290a6f563fe77e5e16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" (UID: \"bfcbf966d484c9290a6f563fe77e5e16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.174285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.174257 4859 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:42:23.174418 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:23.174301 4859 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.308321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.308294 4859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-104.us-west-1.compute.internal" event="NodeReady" Oct 30 21:42:23.308484 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.308376 4859 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:42:23.692768 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.692726 4859 apiserver.go:52] "Watching apiserver" Oct 30 21:42:23.784737 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.784709 4859 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.784905 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.784890 4859 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.845065 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.845024 4859 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:42:23.845236 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:23.845084 4859 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.845236 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.845088 4859 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:42:23.845236 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:23.845170 4859 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal" Oct 30 21:42:23.845497 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.845481 4859 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:42:23.846201 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.846172 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l8jvk","openshift-multus/multus-n5xj4","kube-system/kas-connection-checker-685895c7c9-dq5vk","openshift-image-registry/image-registry-76cdd49c4b-bf2dt","openshift-network-operator/iptables-alerter-xlpzs","openshift-insights/insights-runtime-extractor-r246j","openshift-network-diagnostics/network-check-target-jnj6f","openshift-console-operator/console-operator-7cd56b8dbc-bthdr","openshift-image-registry/image-registry-68687f4c6c-hd5kb","openshift-ingress-canary/ingress-canary-trg4h","openshift-multus/network-metrics-daemon-vbnk5","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27","openshift-insights/insights-operator-7c4986d449-wbpqf","openshift-network-console/networking-console-plugin-86c478bb88-6q9s5","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-104.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-5g5tc","kube-system/kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal","kube-system/kas-connection-checker-685895c7c9-f47zh","openshift-image-registry/node-ca-fgq49","openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw","openshift-ingress/router-default-755f8d7949-984mn","kube-system/konnectivity-agent-lhsfr","openshift-ovn-kubernetes/ovnkube-node-wgpl9","openshift-cluster-node-tuning-operator/tuned-qthxc","openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc","kube-system/kas-connection-checker-685895c7c9-qcvmd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2","openshift-dns/node-resolver-5xm5p","openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr","kube-system/global-pull-secret-syncer-z258k"] Oct 30 21:42:23.847579 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.847552 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:23.848207 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.848189 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:23.849727 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.849709 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:23.850005 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.849986 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:23.850380 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.850363 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:23.852304 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.852284 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:23.859103 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.859074 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:23.860236 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.860218 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:23.860732 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.860716 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:23.901172 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901144 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:42:23.901321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901177 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:42:23.901321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901178 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:42:23.901321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901197 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:23.901321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901234 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:23.901492 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901388 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:23.901740 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901722 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:42:23.901740 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901731 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:23.901740 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901722 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:42:23.901923 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901757 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:42:23.901923 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901773 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:23.901923 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901786 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-t4n5m\"" Oct 30 21:42:23.901923 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901802 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:42:23.901923 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901811 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:23.901923 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901812 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:42:23.902159 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.901936 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:42:23.902159 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902043 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9wg9h\"" Oct 30 21:42:23.902315 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902246 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4t2vs\"" Oct 30 21:42:23.902315 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902263 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:42:23.902315 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902269 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2skfb\"" Oct 30 21:42:23.902315 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902302 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:42:23.902491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902385 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:23.902491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902419 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:42:23.902491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902440 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:42:23.902491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902445 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:42:23.902491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902474 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:42:23.902673 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902506 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:42:23.902673 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902511 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:23.902673 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902511 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:23.902673 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902525 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xlldj\"" Oct 30 21:42:23.902673 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902549 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:23.902673 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902561 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:42:23.902826 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902675 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cgwj9\"" Oct 30 21:42:23.902896 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902879 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:42:23.902988 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902896 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:42:23.902988 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.902898 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:42:23.903107 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.903087 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-btj7g\"" Oct 30 21:42:23.912001 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.911982 4859 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:42:23.949538 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.949480 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:42:23.951506 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:23.951489 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:42:24.001885 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.001851 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:42:24.002066 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.001893 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:42:24.002066 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.001991 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:42:24.002066 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002007 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:42:24.002257 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002133 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:42:24.002257 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002164 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:42:24.002257 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002171 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:42:24.002257 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002208 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:24.002257 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002217 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:42:24.002257 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002212 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:42:24.002257 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002213 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:42:24.002257 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002232 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:42:24.002257 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002243 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:42:24.002556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002263 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zzddg\"" Oct 30 21:42:24.002556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002276 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:42:24.002556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002268 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:42:24.002556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002293 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:42:24.002556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002304 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:42:24.002556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002307 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:42:24.002556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002338 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:24.002556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002338 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:42:24.002556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002358 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:42:24.002935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002624 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:42:24.002935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002732 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4hz4d\"" Oct 30 21:42:24.002935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002764 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:42:24.002935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002852 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p7xlk\"" Oct 30 21:42:24.002935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002875 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:24.002935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002888 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:42:24.002935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002917 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:42:24.002935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002920 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vtv5j\"" Oct 30 21:42:24.003266 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.002962 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sflz7\"" Oct 30 21:42:24.003266 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003083 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:42:24.003266 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003092 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:42:24.003266 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003146 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:42:24.003266 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003158 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:42:24.003266 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003165 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:42:24.003266 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003237 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:42:24.003266 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003265 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003283 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nrkss\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003290 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9csh8\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003303 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003340 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003346 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003348 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003363 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003351 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-msnq6\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003397 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7xfbb\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003397 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003397 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003420 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j7s9f\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003433 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003528 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:42:24.003572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003579 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7fm9b\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003827 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003843 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gscd\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003843 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7tj7w\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003865 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-r22zh\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003871 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pkpl9\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003884 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wcr9d\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003891 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003903 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003926 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zffrj\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.003985 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-82rrg\"" Oct 30 21:42:24.004079 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.004038 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:42:24.004354 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.004168 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:42:24.004354 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.004218 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9qhlv\"" Oct 30 21:42:24.004823 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.004810 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:42:24.007525 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007510 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.007563 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007534 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-trusted-ca\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.007592 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007561 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b21892bd-c791-419a-a1d7-4bdc643b7f07-serving-cert\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.007592 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007578 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-etc-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.007654 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007642 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.007687 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007671 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54q8h\" (UniqueName: \"kubernetes.io/projected/1d29a054-4a50-4709-81c8-01c50df2944e-kube-api-access-54q8h\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.007723 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007690 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6p77\" (UniqueName: \"kubernetes.io/projected/3a85ea4e-16f2-4088-ba73-7e996f46e151-kube-api-access-z6p77\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:42:24.007756 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007722 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45k86\" (UniqueName: \"kubernetes.io/projected/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-api-access-45k86\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.007813 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007794 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b21892bd-c791-419a-a1d7-4bdc643b7f07-serving-cert\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.007883 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007867 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-log-socket\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.007936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007900 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a85ea4e-16f2-4088-ba73-7e996f46e151-host\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:42:24.008024 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007933 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pnln\" (UniqueName: \"kubernetes.io/projected/56e7aa71-e129-4ceb-91a9-06770acc40f7-kube-api-access-8pnln\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.008024 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.007984 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654cq\" (UniqueName: \"kubernetes.io/projected/3bcc5627-b74e-49ed-a82b-36c2920b9917-kube-api-access-654cq\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:24.008024 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008009 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-trusted-ca\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.008024 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008019 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-slash\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.008216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008049 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-system-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.008216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008074 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-multus\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.008216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008106 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-daemon-config\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.008216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008132 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12730f3a-4d15-47f1-af22-9f4e37eedac7-config-volume\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:24.008216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008172 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-device-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.008216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008200 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/641871f0-a0d5-4be1-b2d0-3c7603211724-hosts-file\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:42:24.008437 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008251 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58245\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-kube-api-access-58245\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.008437 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008311 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-socket-dir-parent\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.008437 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008342 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d29a054-4a50-4709-81c8-01c50df2944e-ovn-node-metrics-cert\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.008437 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008410 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-bound-sa-token\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.008606 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008462 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40cea1c9-e71c-4451-926c-398f36e9691c-agent-certs\") pod \"konnectivity-agent-lhsfr\" (UID: \"40cea1c9-e71c-4451-926c-398f36e9691c\") " pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:42:24.008606 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008498 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/81fb1745-4867-438d-b4e0-d44e743e5230-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:24.008606 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008539 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-cni-binary-copy\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.008606 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008574 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-k8s-cni-cncf-io\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.008956 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008301 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12730f3a-4d15-47f1-af22-9f4e37eedac7-config-volume\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:24.009066 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008874 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-multus-certs\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.009117 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009092 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:24.009117 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009094 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d29a054-4a50-4709-81c8-01c50df2944e-ovn-node-metrics-cert\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.009227 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.008939 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-daemon-config\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.009227 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009186 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-image-registry-private-configuration\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.009317 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009232 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-netns\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.009317 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009266 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-run\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.009317 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009270 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a631286-ec63-4b0a-8fc6-f97280e181a1-cni-binary-copy\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.009317 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009239 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40cea1c9-e71c-4451-926c-398f36e9691c-agent-certs\") pod \"konnectivity-agent-lhsfr\" (UID: \"40cea1c9-e71c-4451-926c-398f36e9691c\") " pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:42:24.009452 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009335 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxd6\" (UniqueName: \"kubernetes.io/projected/78532fdc-d35e-4ec3-a48a-18fe0bbe8484-kube-api-access-swxd6\") pod \"kas-connection-checker-685895c7c9-qcvmd\" (UID: \"78532fdc-d35e-4ec3-a48a-18fe0bbe8484\") " pod="kube-system/kas-connection-checker-685895c7c9-qcvmd" Oct 30 21:42:24.009452 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009370 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.009452 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009405 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cgw\" (UniqueName: \"kubernetes.io/projected/12730f3a-4d15-47f1-af22-9f4e37eedac7-kube-api-access-c9cgw\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:24.009452 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009426 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/81fb1745-4867-438d-b4e0-d44e743e5230-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:24.009452 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009437 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cnibin\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.009452 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009442 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-image-registry-private-configuration\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.009706 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009488 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.009706 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009578 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnrn\" (UniqueName: \"kubernetes.io/projected/5387b573-d7bf-47bc-9d20-755835781b5b-kube-api-access-7bnrn\") pod \"network-check-source-6f78d59f99-l2prc\" (UID: \"5387b573-d7bf-47bc-9d20-755835781b5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc" Oct 30 21:42:24.009706 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.009676 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-crio-socket\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.012727 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.012704 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.012834 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.012760 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8lr5\" (UniqueName: \"kubernetes.io/projected/5200dd70-ca15-4547-b344-c772e014e92b-kube-api-access-q8lr5\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:42:24.012901 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.012832 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk7q5\" (UniqueName: \"kubernetes.io/projected/22242d39-b0fc-48a0-a9f2-9a28f7106fbd-kube-api-access-tk7q5\") pod \"volume-data-source-validator-58c9f5f46d-n6b27\" (UID: \"22242d39-b0fc-48a0-a9f2-9a28f7106fbd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27" Oct 30 21:42:24.012940 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.012918 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqcc\" (UniqueName: \"kubernetes.io/projected/61f77d06-9723-4c98-8dd8-80c01d978e2f-kube-api-access-zgqcc\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.013324 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.012974 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-installation-pull-secrets\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.013382 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013357 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfw6m\" (UniqueName: \"kubernetes.io/projected/a749e90b-1748-48e1-b809-89cece3bab0a-kube-api-access-zfw6m\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:24.013426 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013388 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12730f3a-4d15-47f1-af22-9f4e37eedac7-tmp-dir\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:24.013426 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013406 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-ovn\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.013519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013426 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-env-overrides\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.013519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013450 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.013519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013475 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-netns\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.013519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013491 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-installation-pull-secrets\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.013685 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013515 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12730f3a-4d15-47f1-af22-9f4e37eedac7-tmp-dir\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:24.013685 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013592 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.013685 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013613 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-sys-fs\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.013685 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013629 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-env-overrides\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.013685 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013631 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-service-ca-bundle\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.013685 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013665 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-serving-cert\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013735 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40cea1c9-e71c-4451-926c-398f36e9691c-konnectivity-ca\") pod \"konnectivity-agent-lhsfr\" (UID: \"40cea1c9-e71c-4451-926c-398f36e9691c\") " pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013754 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013765 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-serving-cert\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013760 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013769 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-registration-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013785 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21892bd-c791-419a-a1d7-4bdc643b7f07-service-ca-bundle\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013802 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81fb1745-4867-438d-b4e0-d44e743e5230-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013830 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq6m\" (UniqueName: \"kubernetes.io/projected/1a631286-ec63-4b0a-8fc6-f97280e181a1-kube-api-access-7gq6m\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013867 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a85ea4e-16f2-4088-ba73-7e996f46e151-serviceca\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013911 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40cea1c9-e71c-4451-926c-398f36e9691c-konnectivity-ca\") pod \"konnectivity-agent-lhsfr\" (UID: \"40cea1c9-e71c-4451-926c-398f36e9691c\") " pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:42:24.013936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013940 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81fb1745-4867-438d-b4e0-d44e743e5230-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013973 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-var-lib-kubelet\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.013993 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-tmp\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014025 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-dbus\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014079 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-tmp\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014112 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014152 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a85ea4e-16f2-4088-ba73-7e996f46e151-serviceca\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014154 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhwd\" (UniqueName: \"kubernetes.io/projected/aff7cebc-f18e-4b88-bb91-bec7ed9ab097-kube-api-access-wqhwd\") pod \"kas-connection-checker-685895c7c9-f47zh\" (UID: \"aff7cebc-f18e-4b88-bb91-bec7ed9ab097\") " pod="kube-system/kas-connection-checker-685895c7c9-f47zh" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014184 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-script-lib\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014213 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-modprobe-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014236 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-lib-modules\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014275 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-stats-auth\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014314 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-tmp\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014335 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8gcb\" (UniqueName: \"kubernetes.io/projected/5197a548-5e64-49a3-a5b4-421898abe7df-kube-api-access-m8gcb\") pod \"kas-connection-checker-685895c7c9-dq5vk\" (UID: \"5197a548-5e64-49a3-a5b4-421898abe7df\") " pod="kube-system/kas-connection-checker-685895c7c9-dq5vk" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014357 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014379 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3f03966a-97af-46f0-99a5-d50a4f54e744-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014383 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-script-lib\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014407 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-tmp\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.014419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014426 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-socket-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014468 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6307df15-18fe-45cb-b509-298bcb1fcf3b-cert\") pod \"ingress-canary-trg4h\" (UID: \"6307df15-18fe-45cb-b509-298bcb1fcf3b\") " pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014501 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-stats-auth\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014506 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-systemd-units\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014555 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-tuned\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014572 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3f03966a-97af-46f0-99a5-d50a4f54e744-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014577 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3bcc5627-b74e-49ed-a82b-36c2920b9917-config\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014593 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-default-certificate\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014609 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5200dd70-ca15-4547-b344-c772e014e92b-host-slash\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014624 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-cnibin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014626 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-tuned\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014639 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-trusted-ca\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014645 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6307df15-18fe-45cb-b509-298bcb1fcf3b-cert\") pod \"ingress-canary-trg4h\" (UID: \"6307df15-18fe-45cb-b509-298bcb1fcf3b\") " pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014654 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014691 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-certificates\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014767 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-default-certificate\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014790 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014806 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014832 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-var-lib-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014846 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-certificates\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014848 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-bin\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014876 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-trusted-ca\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014910 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014938 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-data-volume\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.014977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014981 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzqbm\" (UniqueName: \"kubernetes.io/projected/b21892bd-c791-419a-a1d7-4bdc643b7f07-kube-api-access-kzqbm\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.014999 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3bcc5627-b74e-49ed-a82b-36c2920b9917-config\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015012 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-bin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015046 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-data-volume\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015055 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015110 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysconfig\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015144 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-certificates\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015171 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015216 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5200dd70-ca15-4547-b344-c772e014e92b-iptables-alerter-script\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015247 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a474868-17e7-4ad9-a3f7-65376fc52940-ca-trust-extracted\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015273 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-bound-sa-token\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015306 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-etc-kubernetes\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015322 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-sys\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015340 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-trusted-ca\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015350 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a474868-17e7-4ad9-a3f7-65376fc52940-ca-trust-extracted\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015360 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015369 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-certificates\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015441 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5200dd70-ca15-4547-b344-c772e014e92b-iptables-alerter-script\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015471 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-host\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015505 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015530 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-systemd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015555 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mt5k\" (UniqueName: \"kubernetes.io/projected/3f03966a-97af-46f0-99a5-d50a4f54e744-kube-api-access-8mt5k\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015573 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-image-registry-private-configuration\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015595 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-installation-pull-secrets\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015617 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-kubelet\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015643 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-conf-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015672 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-config\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015737 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4f9s\" (UniqueName: \"kubernetes.io/projected/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-kube-api-access-j4f9s\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015745 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-trusted-ca\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015761 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015779 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-image-registry-private-configuration\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015791 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txqg8\" (UniqueName: \"kubernetes.io/projected/65c914e8-45f5-4219-9790-b5fc849d39f7-kube-api-access-txqg8\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015780 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-installation-pull-secrets\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015820 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7l4k\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-kube-api-access-t7l4k\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015848 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-config\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015867 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-config\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015874 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015905 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015939 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641871f0-a0d5-4be1-b2d0-3c7603211724-tmp-dir\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.015980 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqm4k\" (UniqueName: \"kubernetes.io/projected/641871f0-a0d5-4be1-b2d0-3c7603211724-kube-api-access-gqm4k\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:42:24.015997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016013 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-kubelet\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016035 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641871f0-a0d5-4be1-b2d0-3c7603211724-tmp-dir\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016040 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3bcc5627-b74e-49ed-a82b-36c2920b9917-serving-cert\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016070 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d29a054-4a50-4709-81c8-01c50df2944e-ovnkube-config\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016097 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbcw8\" (UniqueName: \"kubernetes.io/projected/6307df15-18fe-45cb-b509-298bcb1fcf3b-kube-api-access-tbcw8\") pod \"ingress-canary-trg4h\" (UID: \"6307df15-18fe-45cb-b509-298bcb1fcf3b\") " pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016113 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016142 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016117 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016188 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3bcc5627-b74e-49ed-a82b-36c2920b9917-serving-cert\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016190 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-conf\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016220 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-system-cni-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016265 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-netd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016288 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-kubernetes\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016330 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016352 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-snapshots\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016367 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-binary-copy\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016383 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016404 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016439 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016464 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-kubelet-config\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016490 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016519 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-hostroot\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016539 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg55q\" (UniqueName: \"kubernetes.io/projected/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-kube-api-access-dg55q\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016555 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-node-log\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016557 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b21892bd-c791-419a-a1d7-4bdc643b7f07-snapshots\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016569 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-systemd\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016578 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cni-binary-copy\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016592 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/699c93ce-4424-4460-8cb2-b5db6ca24033-ca-trust-extracted\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016611 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgn5s\" (UniqueName: \"kubernetes.io/projected/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kube-api-access-lgn5s\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016632 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvjt\" (UniqueName: \"kubernetes.io/projected/81fb1745-4867-438d-b4e0-d44e743e5230-kube-api-access-4rvjt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016752 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/699c93ce-4424-4460-8cb2-b5db6ca24033-ca-trust-extracted\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.017285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.016753 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70fbe0ee-70e4-47b1-8094-4faa96292c6b-original-pull-secret\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:24.063563 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.063525 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45k86\" (UniqueName: \"kubernetes.io/projected/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-api-access-45k86\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.063722 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.063703 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-bound-sa-token\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.064513 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.064481 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnrn\" (UniqueName: \"kubernetes.io/projected/5387b573-d7bf-47bc-9d20-755835781b5b-kube-api-access-7bnrn\") pod \"network-check-source-6f78d59f99-l2prc\" (UID: \"5387b573-d7bf-47bc-9d20-755835781b5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2prc" Oct 30 21:42:24.064668 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.064546 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58245\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-kube-api-access-58245\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.064668 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.064601 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxd6\" (UniqueName: \"kubernetes.io/projected/78532fdc-d35e-4ec3-a48a-18fe0bbe8484-kube-api-access-swxd6\") pod \"kas-connection-checker-685895c7c9-qcvmd\" (UID: \"78532fdc-d35e-4ec3-a48a-18fe0bbe8484\") " pod="kube-system/kas-connection-checker-685895c7c9-qcvmd" Oct 30 21:42:24.064668 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.064629 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6p77\" (UniqueName: \"kubernetes.io/projected/3a85ea4e-16f2-4088-ba73-7e996f46e151-kube-api-access-z6p77\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:42:24.064868 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.064851 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cgw\" (UniqueName: \"kubernetes.io/projected/12730f3a-4d15-47f1-af22-9f4e37eedac7-kube-api-access-c9cgw\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:24.064923 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.064903 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54q8h\" (UniqueName: \"kubernetes.io/projected/1d29a054-4a50-4709-81c8-01c50df2944e-kube-api-access-54q8h\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.065121 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.065100 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654cq\" (UniqueName: \"kubernetes.io/projected/3bcc5627-b74e-49ed-a82b-36c2920b9917-kube-api-access-654cq\") pod \"service-ca-operator-544d749c8c-k2gbw\" (UID: \"3bcc5627-b74e-49ed-a82b-36c2920b9917\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" Oct 30 21:42:24.065306 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.065291 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pnln\" (UniqueName: \"kubernetes.io/projected/56e7aa71-e129-4ceb-91a9-06770acc40f7-kube-api-access-8pnln\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.067539 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.067517 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8lr5\" (UniqueName: \"kubernetes.io/projected/5200dd70-ca15-4547-b344-c772e014e92b-kube-api-access-q8lr5\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:42:24.072014 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.071993 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gq6m\" (UniqueName: \"kubernetes.io/projected/1a631286-ec63-4b0a-8fc6-f97280e181a1-kube-api-access-7gq6m\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.072086 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.072000 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqcc\" (UniqueName: \"kubernetes.io/projected/61f77d06-9723-4c98-8dd8-80c01d978e2f-kube-api-access-zgqcc\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.072086 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.072069 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfw6m\" (UniqueName: \"kubernetes.io/projected/a749e90b-1748-48e1-b809-89cece3bab0a-kube-api-access-zfw6m\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:24.072399 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.072383 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk7q5\" (UniqueName: \"kubernetes.io/projected/22242d39-b0fc-48a0-a9f2-9a28f7106fbd-kube-api-access-tk7q5\") pod \"volume-data-source-validator-58c9f5f46d-n6b27\" (UID: \"22242d39-b0fc-48a0-a9f2-9a28f7106fbd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n6b27" Oct 30 21:42:24.073373 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.073351 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-bound-sa-token\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.074999 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.074976 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8gcb\" (UniqueName: \"kubernetes.io/projected/5197a548-5e64-49a3-a5b4-421898abe7df-kube-api-access-m8gcb\") pod \"kas-connection-checker-685895c7c9-dq5vk\" (UID: \"5197a548-5e64-49a3-a5b4-421898abe7df\") " pod="kube-system/kas-connection-checker-685895c7c9-dq5vk" Oct 30 21:42:24.075109 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.075035 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4f9s\" (UniqueName: \"kubernetes.io/projected/f9d15d9a-71ac-4572-98ca-c5f5df251cb2-kube-api-access-j4f9s\") pod \"console-operator-7cd56b8dbc-bthdr\" (UID: \"f9d15d9a-71ac-4572-98ca-c5f5df251cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:24.075230 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.075207 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7l4k\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-kube-api-access-t7l4k\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.075464 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.075440 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txqg8\" (UniqueName: \"kubernetes.io/projected/65c914e8-45f5-4219-9790-b5fc849d39f7-kube-api-access-txqg8\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:24.075526 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.075507 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqm4k\" (UniqueName: \"kubernetes.io/projected/641871f0-a0d5-4be1-b2d0-3c7603211724-kube-api-access-gqm4k\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:42:24.075641 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.075622 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mt5k\" (UniqueName: \"kubernetes.io/projected/3f03966a-97af-46f0-99a5-d50a4f54e744-kube-api-access-8mt5k\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:24.075705 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.075653 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgn5s\" (UniqueName: \"kubernetes.io/projected/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kube-api-access-lgn5s\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.075992 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.075971 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzqbm\" (UniqueName: \"kubernetes.io/projected/b21892bd-c791-419a-a1d7-4bdc643b7f07-kube-api-access-kzqbm\") pod \"insights-operator-7c4986d449-wbpqf\" (UID: \"b21892bd-c791-419a-a1d7-4bdc643b7f07\") " pod="openshift-insights/insights-operator-7c4986d449-wbpqf" Oct 30 21:42:24.076059 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.075992 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfdbg\" (UniqueName: \"kubernetes.io/projected/a403f7b4-a616-4807-8263-678045389f84-kube-api-access-gfdbg\") pod \"network-check-target-jnj6f\" (UID: \"a403f7b4-a616-4807-8263-678045389f84\") " pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:24.076059 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.076013 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg55q\" (UniqueName: \"kubernetes.io/projected/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-kube-api-access-dg55q\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.076059 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.076049 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbcw8\" (UniqueName: \"kubernetes.io/projected/6307df15-18fe-45cb-b509-298bcb1fcf3b-kube-api-access-tbcw8\") pod \"ingress-canary-trg4h\" (UID: \"6307df15-18fe-45cb-b509-298bcb1fcf3b\") " pod="openshift-ingress-canary/ingress-canary-trg4h" Oct 30 21:42:24.076199 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.076117 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhwd\" (UniqueName: \"kubernetes.io/projected/aff7cebc-f18e-4b88-bb91-bec7ed9ab097-kube-api-access-wqhwd\") pod \"kas-connection-checker-685895c7c9-f47zh\" (UID: \"aff7cebc-f18e-4b88-bb91-bec7ed9ab097\") " pod="kube-system/kas-connection-checker-685895c7c9-f47zh" Oct 30 21:42:24.076527 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.076511 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvjt\" (UniqueName: \"kubernetes.io/projected/81fb1745-4867-438d-b4e0-d44e743e5230-kube-api-access-4rvjt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q4vks\" (UID: \"81fb1745-4867-438d-b4e0-d44e743e5230\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" Oct 30 21:42:24.117419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117386 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-slash\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.117419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117423 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-system-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117444 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-multus\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117466 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-device-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117485 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/641871f0-a0d5-4be1-b2d0-3c7603211724-hosts-file\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117493 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-slash\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117512 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-socket-dir-parent\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117522 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-system-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117523 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-multus\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117536 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-device-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117552 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/641871f0-a0d5-4be1-b2d0-3c7603211724-hosts-file\") pod \"node-resolver-5xm5p\" (UID: \"641871f0-a0d5-4be1-b2d0-3c7603211724\") " pod="openshift-dns/node-resolver-5xm5p" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117539 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-k8s-cni-cncf-io\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117575 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-socket-dir-parent\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.117598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117583 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-multus-certs\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117606 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117611 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-multus-certs\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117577 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-k8s-cni-cncf-io\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117625 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-netns\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117641 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-run\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117666 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-run-netns\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117682 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117698 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cnibin\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117712 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117730 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-crio-socket\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117745 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.117751 4859 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117771 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-ovn\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117779 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-run\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117781 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-cni-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117819 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-crio-socket\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117829 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-netns\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117834 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-netns\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117839 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-ovn\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117829 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-cnibin\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.117836 4859 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117867 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117888 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-sys-fs\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.117908 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.617887592 +0000 UTC m=+1.990514321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.117918 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-sys-fs\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.117967 4859 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.117968 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.617932101 +0000 UTC m=+1.990558832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : configmap "kube-rbac-proxy" not found Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118000 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.617989733 +0000 UTC m=+1.990616465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : secret "router-metrics-certs-default" not found Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118020 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls podName:12730f3a-4d15-47f1-af22-9f4e37eedac7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.618011085 +0000 UTC m=+1.990637817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls") pod "dns-default-l8jvk" (UID: "12730f3a-4d15-47f1-af22-9f4e37eedac7") : secret "dns-default-metrics-tls" not found Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118049 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118082 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-registration-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118109 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-var-lib-kubelet\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118133 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-dbus\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118142 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118146 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-registration-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118159 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118181 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-var-lib-kubelet\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118185 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-modprobe-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.118181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118194 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-dbus\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118215 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-lib-modules\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118240 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118248 4859 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118258 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-modprobe-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118266 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-socket-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118291 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls podName:65c914e8-45f5-4219-9790-b5fc849d39f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.618275703 +0000 UTC m=+1.990902429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jhjqr" (UID: "65c914e8-45f5-4219-9790-b5fc849d39f7") : secret "samples-operator-tls" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118303 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-lib-modules\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118320 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-systemd-units\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118337 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-d\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118322 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-socket-dir\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118351 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5200dd70-ca15-4547-b344-c772e014e92b-host-slash\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118374 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5200dd70-ca15-4547-b344-c772e014e92b-host-slash\") pod \"iptables-alerter-xlpzs\" (UID: \"5200dd70-ca15-4547-b344-c772e014e92b\") " pod="openshift-network-operator/iptables-alerter-xlpzs" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118352 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-systemd-units\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118374 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-cnibin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118403 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118411 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-cnibin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118426 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118441 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-var-lib-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118455 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-bin\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118480 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-bin\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118484 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118502 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-var-lib-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118516 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-bin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118517 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118522 4859 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118556 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-cni-bin\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118567 4859 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68687f4c6c-hd5kb: secret "image-registry-tls" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118548 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysconfig\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118592 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysconfig\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118599 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118620 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-etc-kubernetes\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118568 4859 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118628 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls podName:699c93ce-4424-4460-8cb2-b5db6ca24033 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.618618499 +0000 UTC m=+1.991245255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls") pod "image-registry-68687f4c6c-hd5kb" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033") : secret "image-registry-tls" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118661 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-etc-kubernetes\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118674 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls podName:3f03966a-97af-46f0-99a5-d50a4f54e744 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.618665404 +0000 UTC m=+1.991292128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xmt2j" (UID: "3f03966a-97af-46f0-99a5-d50a4f54e744") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118683 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-sys\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118694 4859 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118703 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-host\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118726 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-systemd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.118737 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert podName:7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.618723134 +0000 UTC m=+1.991349876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q9s5" (UID: "7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a") : secret "networking-console-plugin-cert" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118743 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-host\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118758 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-systemd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118758 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-sys\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118770 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-kubelet\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118798 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-conf-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118826 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-kubelet\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118851 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118874 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-kubelet\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118874 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-multus-conf-dir\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118884 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118851 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-host-var-lib-kubelet\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118873 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-conf\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118921 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-system-cni-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118938 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-netd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118974 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-kubernetes\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118978 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0-system-cni-dir\") pod \"multus-additional-cni-plugins-5g5tc\" (UID: \"7e4b501f-ff9f-4ed7-9b82-68d5fd929cf0\") " pod="openshift-multus/multus-additional-cni-plugins-5g5tc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118979 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-cni-netd\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118998 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119020 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119037 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.118984 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-sysctl-conf\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119054 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119086 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-kubelet-config\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119078 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119112 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-hostroot\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119114 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-kubernetes\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.119118 4859 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119139 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a631286-ec63-4b0a-8fc6-f97280e181a1-hostroot\") pod \"multus-n5xj4\" (UID: \"1a631286-ec63-4b0a-8fc6-f97280e181a1\") " pod="openshift-multus/multus-n5xj4" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119142 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-node-log\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119103 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5528ca3c-d24a-485c-b896-54fdc2fdb5ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-vczx2\" (UID: \"5528ca3c-d24a-485c-b896-54fdc2fdb5ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vczx2" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119149 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70fbe0ee-70e4-47b1-8094-4faa96292c6b-kubelet-config\") pod \"global-pull-secret-syncer-z258k\" (UID: \"70fbe0ee-70e4-47b1-8094-4faa96292c6b\") " pod="kube-system/global-pull-secret-syncer-z258k" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.119165 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.619153167 +0000 UTC m=+1.991779895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : secret "metrics-daemon-secret" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.119169 4859 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119179 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-node-log\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.119200 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.619190625 +0000 UTC m=+1.991817376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119215 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-systemd\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119180 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61f77d06-9723-4c98-8dd8-80c01d978e2f-etc-systemd\") pod \"tuned-qthxc\" (UID: \"61f77d06-9723-4c98-8dd8-80c01d978e2f\") " pod="openshift-cluster-node-tuning-operator/tuned-qthxc" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119260 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119284 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-etc-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119299 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119319 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-log-socket\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119342 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a85ea4e-16f2-4088-ba73-7e996f46e151-host\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119352 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-etc-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119372 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-run-openvswitch\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119379 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d29a054-4a50-4709-81c8-01c50df2944e-log-socket\") pod \"ovnkube-node-wgpl9\" (UID: \"1d29a054-4a50-4709-81c8-01c50df2944e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.119394 4859 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.119398 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a85ea4e-16f2-4088-ba73-7e996f46e151-host\") pod \"node-ca-fgq49\" (UID: \"3a85ea4e-16f2-4088-ba73-7e996f46e151\") " pod="openshift-image-registry/node-ca-fgq49" Oct 30 21:42:24.119485 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.119406 4859 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd49c4b-bf2dt: secret "image-registry-tls" not found Oct 30 21:42:24.121847 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.119433 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls podName:8a474868-17e7-4ad9-a3f7-65376fc52940 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.619424613 +0000 UTC m=+1.992051338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls") pod "image-registry-76cdd49c4b-bf2dt" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940") : secret "image-registry-tls" not found Oct 30 21:42:24.148050 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.148024 4859 scope.go:122] "RemoveContainer" containerID="328a8eaab798437384f355cbcbec523ecb9f13793741b3dada3d785415388da3" Oct 30 21:42:24.624272 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624238 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:24.624272 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624278 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624300 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624341 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624366 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624391 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624393 4859 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624437 4859 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624456 4859 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624466 4859 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624398 4859 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624484 4859 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:24.624500 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624499 4859 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd49c4b-bf2dt: secret "image-registry-tls" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624488 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls podName:3f03966a-97af-46f0-99a5-d50a4f54e744 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624470826 +0000 UTC m=+2.997097556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xmt2j" (UID: "3f03966a-97af-46f0-99a5-d50a4f54e744") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624554 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert podName:7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624540056 +0000 UTC m=+2.997166782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q9s5" (UID: "7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a") : secret "networking-console-plugin-cert" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624488 4859 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68687f4c6c-hd5kb: secret "image-registry-tls" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624568 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624560535 +0000 UTC m=+2.997187260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624585 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls podName:699c93ce-4424-4460-8cb2-b5db6ca24033 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624575649 +0000 UTC m=+2.997202374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls") pod "image-registry-68687f4c6c-hd5kb" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033") : secret "image-registry-tls" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624598 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls podName:8a474868-17e7-4ad9-a3f7-65376fc52940 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624592741 +0000 UTC m=+2.997219466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls") pod "image-registry-76cdd49c4b-bf2dt" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940") : secret "image-registry-tls" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624608 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624603335 +0000 UTC m=+2.997230060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : secret "metrics-daemon-secret" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624631 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624657 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624676 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624705 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.624729 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624741 4859 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624749 4859 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624770 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624762836 +0000 UTC m=+2.997389561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : configmap "kube-rbac-proxy" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624784 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls podName:12730f3a-4d15-47f1-af22-9f4e37eedac7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624775541 +0000 UTC m=+2.997402270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls") pod "dns-default-l8jvk" (UID: "12730f3a-4d15-47f1-af22-9f4e37eedac7") : secret "dns-default-metrics-tls" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624798 4859 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624805 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624794944 +0000 UTC m=+2.997421672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624822 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls podName:65c914e8-45f5-4219-9790-b5fc849d39f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624813774 +0000 UTC m=+2.997440540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jhjqr" (UID: "65c914e8-45f5-4219-9790-b5fc849d39f7") : secret "samples-operator-tls" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624827 4859 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:24.624854 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.624851 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.624843613 +0000 UTC m=+2.997470370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : secret "router-metrics-certs-default" not found Oct 30 21:42:24.791786 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.791759 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 21:42:24.792215 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.792154 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/1.log" Oct 30 21:42:24.792215 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.792188 4859 generic.go:356] "Generic (PLEG): container finished" podID="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" containerID="424669f95ef6a9c7411b8f3d59884976a899774c7a75064a63fdb5762f5e8ffa" exitCode=255 Oct 30 21:42:24.792301 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.792277 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" event={"ID":"f9d15d9a-71ac-4572-98ca-c5f5df251cb2","Type":"ContainerDied","Data":"424669f95ef6a9c7411b8f3d59884976a899774c7a75064a63fdb5762f5e8ffa"} Oct 30 21:42:24.792359 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.792345 4859 scope.go:122] "RemoveContainer" containerID="328a8eaab798437384f355cbcbec523ecb9f13793741b3dada3d785415388da3" Oct 30 21:42:24.792831 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:24.792814 4859 scope.go:122] "RemoveContainer" containerID="424669f95ef6a9c7411b8f3d59884976a899774c7a75064a63fdb5762f5e8ffa" Oct 30 21:42:24.793029 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:24.793011 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthdr_openshift-console-operator(f9d15d9a-71ac-4572-98ca-c5f5df251cb2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" podUID="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" Oct 30 21:42:25.286841 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.286813 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:25.635728 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.635692 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:25.635728 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.635734 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.635756 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.635788 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.635808 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.635828 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.635844 4859 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.635895 4859 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.635905 4859 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd49c4b-bf2dt: secret "image-registry-tls" not found Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.635905 4859 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.635848 4859 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.635927 4859 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.635994 4859 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.635912 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls podName:3f03966a-97af-46f0-99a5-d50a4f54e744 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.635898143 +0000 UTC m=+5.008524868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xmt2j" (UID: "3f03966a-97af-46f0-99a5-d50a4f54e744") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:25.636021 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.635930 4859 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68687f4c6c-hd5kb: secret "image-registry-tls" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636030 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls podName:8a474868-17e7-4ad9-a3f7-65376fc52940 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.63600935 +0000 UTC m=+5.008636079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls") pod "image-registry-76cdd49c4b-bf2dt" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940") : secret "image-registry-tls" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636046 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert podName:7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.636038649 +0000 UTC m=+5.008665378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q9s5" (UID: "7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a") : secret "networking-console-plugin-cert" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636058 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.636052183 +0000 UTC m=+5.008678907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.636077 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.636110 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636115 4859 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.636130 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636148 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls podName:12730f3a-4d15-47f1-af22-9f4e37eedac7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.636133437 +0000 UTC m=+5.008760164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls") pod "dns-default-l8jvk" (UID: "12730f3a-4d15-47f1-af22-9f4e37eedac7") : secret "dns-default-metrics-tls" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636168 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.636161618 +0000 UTC m=+5.008788343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : secret "metrics-daemon-secret" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636178 4859 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636184 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls podName:699c93ce-4424-4460-8cb2-b5db6ca24033 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.636178235 +0000 UTC m=+5.008804960 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls") pod "image-registry-68687f4c6c-hd5kb" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033") : secret "image-registry-tls" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636209 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.636194718 +0000 UTC m=+5.008821456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636226 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.636219529 +0000 UTC m=+5.008846258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : configmap "kube-rbac-proxy" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.636249 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.636280 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636356 4859 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:25.636378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636358 4859 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:25.636850 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636389 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls podName:65c914e8-45f5-4219-9790-b5fc849d39f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.636378664 +0000 UTC m=+5.009005390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jhjqr" (UID: "65c914e8-45f5-4219-9790-b5fc849d39f7") : secret "samples-operator-tls" not found Oct 30 21:42:25.636850 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.636402 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.63639663 +0000 UTC m=+5.009023354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : secret "router-metrics-certs-default" not found Oct 30 21:42:25.741757 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.741726 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:25.797493 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.797458 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 21:42:25.797974 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.797655 4859 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:25.994979 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.994872 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:25.995255 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:25.995241 4859 scope.go:122] "RemoveContainer" containerID="424669f95ef6a9c7411b8f3d59884976a899774c7a75064a63fdb5762f5e8ffa" Oct 30 21:42:25.995434 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:25.995417 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthdr_openshift-console-operator(f9d15d9a-71ac-4572-98ca-c5f5df251cb2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" podUID="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" Oct 30 21:42:26.492238 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:26.492208 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5xm5p_641871f0-a0d5-4be1-b2d0-3c7603211724/dns-node-resolver/0.log" Oct 30 21:42:26.577899 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:26.577868 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgpl9" Oct 30 21:42:27.282979 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.282929 4859 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhsfr_40cea1c9-e71c-4451-926c-398f36e9691c/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:27.492494 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.492464 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fgq49_3a85ea4e-16f2-4088-ba73-7e996f46e151/node-ca/0.log" Oct 30 21:42:27.524494 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.524467 4859 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhsfr_40cea1c9-e71c-4451-926c-398f36e9691c/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:27.654157 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654121 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:27.654308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654165 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:27.654308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654188 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:27.654308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654210 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:27.654308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654248 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:27.654308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654269 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:27.654308 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654284 4859 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654362 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls podName:65c914e8-45f5-4219-9790-b5fc849d39f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.654344142 +0000 UTC m=+9.026970880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jhjqr" (UID: "65c914e8-45f5-4219-9790-b5fc849d39f7") : secret "samples-operator-tls" not found Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654385 4859 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654290 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654418 4859 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654393 4859 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654447 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs podName:a749e90b-1748-48e1-b809-89cece3bab0a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.654429862 +0000 UTC m=+9.027056605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs") pod "network-metrics-daemon-vbnk5" (UID: "a749e90b-1748-48e1-b809-89cece3bab0a") : secret "metrics-daemon-secret" not found Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654485 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.654469926 +0000 UTC m=+9.027096673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654503 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert podName:7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.654493997 +0000 UTC m=+9.027120725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q9s5" (UID: "7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a") : secret "networking-console-plugin-cert" not found Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654523 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654562 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654567 4859 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:42:27.654589 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654579 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:27.655037 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.654608 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:27.655037 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654670 4859 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:27.655037 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654697 4859 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:27.655037 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654718 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.654699167 +0000 UTC m=+9.027325906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:27.655037 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654674 4859 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:27.655037 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654741 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.654730702 +0000 UTC m=+9.027357431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : secret "router-metrics-certs-default" not found Oct 30 21:42:27.655037 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654763 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.654746818 +0000 UTC m=+9.027373552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : configmap "kube-rbac-proxy" not found Oct 30 21:42:27.655037 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:27.654780 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls podName:12730f3a-4d15-47f1-af22-9f4e37eedac7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.65477286 +0000 UTC m=+9.027399588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls") pod "dns-default-l8jvk" (UID: "12730f3a-4d15-47f1-af22-9f4e37eedac7") : secret "dns-default-metrics-tls" not found Oct 30 21:42:27.656824 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.656804 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f03966a-97af-46f0-99a5-d50a4f54e744-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xmt2j\" (UID: \"3f03966a-97af-46f0-99a5-d50a4f54e744\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:27.656965 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.656930 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"image-registry-76cdd49c4b-bf2dt\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:27.657007 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.656930 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"image-registry-68687f4c6c-hd5kb\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:27.750203 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.750161 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" Oct 30 21:42:27.751286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.751257 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:27.759520 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:27.759494 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:28.134239 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.134204 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j"] Oct 30 21:42:28.170270 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:28.170242 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f03966a_97af_46f0_99a5_d50a4f54e744.slice/crio-5e101c6e1e9f1d4702a6755b661282dcd8a21285314a4c6bb4db5cbba5e98333 WatchSource:0}: Error finding container 5e101c6e1e9f1d4702a6755b661282dcd8a21285314a4c6bb4db5cbba5e98333: Status 404 returned error can't find the container with id 5e101c6e1e9f1d4702a6755b661282dcd8a21285314a4c6bb4db5cbba5e98333 Oct 30 21:42:28.172311 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.172294 4859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:42:28.195390 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.195358 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68687f4c6c-hd5kb"] Oct 30 21:42:28.198582 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:28.198555 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699c93ce_4424_4460_8cb2_b5db6ca24033.slice/crio-283f0315a90b3bb0b2ff29e2ba46bd636db7e1c8cbc8b63564c1495080430dcf WatchSource:0}: Error finding container 283f0315a90b3bb0b2ff29e2ba46bd636db7e1c8cbc8b63564c1495080430dcf: Status 404 returned error can't find the container with id 283f0315a90b3bb0b2ff29e2ba46bd636db7e1c8cbc8b63564c1495080430dcf Oct 30 21:42:28.233626 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.233507 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76cdd49c4b-bf2dt"] Oct 30 21:42:28.237248 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:28.237224 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a474868_17e7_4ad9_a3f7_65376fc52940.slice/crio-c1c1b1816d545ea0b2ad686f7b56bb59901c3c51b13e04cb8d16061f9ceec800 WatchSource:0}: Error finding container c1c1b1816d545ea0b2ad686f7b56bb59901c3c51b13e04cb8d16061f9ceec800: Status 404 returned error can't find the container with id c1c1b1816d545ea0b2ad686f7b56bb59901c3c51b13e04cb8d16061f9ceec800 Oct 30 21:42:28.691858 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.691830 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-trg4h_6307df15-18fe-45cb-b509-298bcb1fcf3b/serve-healthcheck-canary/0.log" Oct 30 21:42:28.764822 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.764787 4859 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:28.765203 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.765190 4859 scope.go:122] "RemoveContainer" containerID="424669f95ef6a9c7411b8f3d59884976a899774c7a75064a63fdb5762f5e8ffa" Oct 30 21:42:28.765369 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:28.765351 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthdr_openshift-console-operator(f9d15d9a-71ac-4572-98ca-c5f5df251cb2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" podUID="f9d15d9a-71ac-4572-98ca-c5f5df251cb2" Oct 30 21:42:28.818331 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.818297 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" event={"ID":"8a474868-17e7-4ad9-a3f7-65376fc52940","Type":"ContainerStarted","Data":"d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1"} Oct 30 21:42:28.818514 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.818352 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" event={"ID":"8a474868-17e7-4ad9-a3f7-65376fc52940","Type":"ContainerStarted","Data":"c1c1b1816d545ea0b2ad686f7b56bb59901c3c51b13e04cb8d16061f9ceec800"} Oct 30 21:42:28.819844 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.819820 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" event={"ID":"699c93ce-4424-4460-8cb2-b5db6ca24033","Type":"ContainerStarted","Data":"613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7"} Oct 30 21:42:28.819940 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.819858 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" event={"ID":"699c93ce-4424-4460-8cb2-b5db6ca24033","Type":"ContainerStarted","Data":"283f0315a90b3bb0b2ff29e2ba46bd636db7e1c8cbc8b63564c1495080430dcf"} Oct 30 21:42:28.821159 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.821141 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" event={"ID":"3f03966a-97af-46f0-99a5-d50a4f54e744","Type":"ContainerStarted","Data":"5e101c6e1e9f1d4702a6755b661282dcd8a21285314a4c6bb4db5cbba5e98333"} Oct 30 21:42:28.897845 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:28.897816 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lhsfr" Oct 30 21:42:29.294504 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:29.294466 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-q4vks_81fb1745-4867-438d-b4e0-d44e743e5230/kube-storage-version-migrator-operator/0.log" Oct 30 21:42:29.824022 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:29.823993 4859 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:29.824498 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:29.823993 4859 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:30.829354 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:30.829320 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xmt2j" event={"ID":"3f03966a-97af-46f0-99a5-d50a4f54e744","Type":"ContainerStarted","Data":"36fb0c81e692a6a01a1eba93d14676ebf1de34cf9a3249405c8890bde3ee7f6d"} Oct 30 21:42:31.695267 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.695222 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:31.695491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.695282 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:31.695491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.695324 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:31.695491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.695364 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:31.695491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.695426 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:31.695491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.695456 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:31.695491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.695481 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:31.695848 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.695513 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:31.695848 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:31.695584 4859 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:31.695848 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:31.695617 4859 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:31.695848 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:31.695628 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:39.69560599 +0000 UTC m=+17.068232731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:31.695848 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:31.695652 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:39.695641893 +0000 UTC m=+17.068268619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:31.695848 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:31.695665 4859 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:31.695848 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:31.695668 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs podName:56e7aa71-e129-4ceb-91a9-06770acc40f7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:39.695658295 +0000 UTC m=+17.068285020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs") pod "router-default-755f8d7949-984mn" (UID: "56e7aa71-e129-4ceb-91a9-06770acc40f7") : secret "router-metrics-certs-default" not found Oct 30 21:42:31.695848 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:42:31.695697 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm podName:b2c6a540-c40a-4a1e-b7c6-70e4384f1f24 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:39.695686464 +0000 UTC m=+17.068313194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r246j" (UID: "b2c6a540-c40a-4a1e-b7c6-70e4384f1f24") : configmap "kube-rbac-proxy" not found Oct 30 21:42:31.697858 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.697823 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12730f3a-4d15-47f1-af22-9f4e37eedac7-metrics-tls\") pod \"dns-default-l8jvk\" (UID: \"12730f3a-4d15-47f1-af22-9f4e37eedac7\") " pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:31.698049 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.698028 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q9s5\" (UID: \"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:31.698190 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.698167 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a749e90b-1748-48e1-b809-89cece3bab0a-metrics-certs\") pod \"network-metrics-daemon-vbnk5\" (UID: \"a749e90b-1748-48e1-b809-89cece3bab0a\") " pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:31.698285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.698270 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65c914e8-45f5-4219-9790-b5fc849d39f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jhjqr\" (UID: \"65c914e8-45f5-4219-9790-b5fc849d39f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:31.949085 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.948998 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" Oct 30 21:42:31.951139 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.951107 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" Oct 30 21:42:31.953240 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.953222 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbnk5" Oct 30 21:42:31.961426 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:31.961400 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:32.346770 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:32.346736 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr"] Oct 30 21:42:32.396576 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:32.396178 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6q9s5"] Oct 30 21:42:32.400460 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:32.400432 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f54c5d5_136a_40f5_8e8d_5eb3b35daf9a.slice/crio-3e95b4aa0ae2af93b981658247af171bd6e80ce4b1ff887c7bcf7a84a1417a9c WatchSource:0}: Error finding container 3e95b4aa0ae2af93b981658247af171bd6e80ce4b1ff887c7bcf7a84a1417a9c: Status 404 returned error can't find the container with id 3e95b4aa0ae2af93b981658247af171bd6e80ce4b1ff887c7bcf7a84a1417a9c Oct 30 21:42:32.453925 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:32.453895 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l8jvk"] Oct 30 21:42:32.456893 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:32.456845 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12730f3a_4d15_47f1_af22_9f4e37eedac7.slice/crio-33548a2b6f3199775078396641a988b3f455663107e4123725ca21b386ae2ab3 WatchSource:0}: Error finding container 33548a2b6f3199775078396641a988b3f455663107e4123725ca21b386ae2ab3: Status 404 returned error can't find the container with id 33548a2b6f3199775078396641a988b3f455663107e4123725ca21b386ae2ab3 Oct 30 21:42:32.457622 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:32.457598 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vbnk5"] Oct 30 21:42:32.460562 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:32.460543 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda749e90b_1748_48e1_b809_89cece3bab0a.slice/crio-d41a1465c2e5352cabd3662f26900d6bc04533052c94e323185d77d3ae6bbe05 WatchSource:0}: Error finding container d41a1465c2e5352cabd3662f26900d6bc04533052c94e323185d77d3ae6bbe05: Status 404 returned error can't find the container with id d41a1465c2e5352cabd3662f26900d6bc04533052c94e323185d77d3ae6bbe05 Oct 30 21:42:32.836770 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:32.836683 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vbnk5" event={"ID":"a749e90b-1748-48e1-b809-89cece3bab0a","Type":"ContainerStarted","Data":"d41a1465c2e5352cabd3662f26900d6bc04533052c94e323185d77d3ae6bbe05"} Oct 30 21:42:32.837793 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:32.837761 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l8jvk" event={"ID":"12730f3a-4d15-47f1-af22-9f4e37eedac7","Type":"ContainerStarted","Data":"33548a2b6f3199775078396641a988b3f455663107e4123725ca21b386ae2ab3"} Oct 30 21:42:32.839080 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:32.839055 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" event={"ID":"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a","Type":"ContainerStarted","Data":"3e95b4aa0ae2af93b981658247af171bd6e80ce4b1ff887c7bcf7a84a1417a9c"} Oct 30 21:42:32.840161 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:32.840138 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" event={"ID":"65c914e8-45f5-4219-9790-b5fc849d39f7","Type":"ContainerStarted","Data":"bbccd2b4ea71f06bb321063d735f00c2f66bfe32e02f1e5732fb9b6f8da1f520"} Oct 30 21:42:33.098676 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:33.098641 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:33.120400 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:33.120368 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:33.152072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:33.152036 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:34.734264 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:34.734226 4859 scope.go:122] "RemoveContainer" containerID="424669f95ef6a9c7411b8f3d59884976a899774c7a75064a63fdb5762f5e8ffa" Oct 30 21:42:35.856574 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:35.856542 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vbnk5" event={"ID":"a749e90b-1748-48e1-b809-89cece3bab0a","Type":"ContainerStarted","Data":"b97516f7ec32b4b832f5940aed24331036aa8b1bffb16e14e4fe97d35a3eb260"} Oct 30 21:42:35.857039 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:35.856608 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vbnk5" event={"ID":"a749e90b-1748-48e1-b809-89cece3bab0a","Type":"ContainerStarted","Data":"8ba2a0bd51f1b9dbec02c673fda885776262cbc14a6b664b07f01c4f0e720c2c"} Oct 30 21:42:35.859889 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:35.859860 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6q9s5" event={"ID":"7f54c5d5-136a-40f5-8e8d-5eb3b35daf9a","Type":"ContainerStarted","Data":"2b17d7aa6844507beaecafa80879dea2a5df9e17b182daaf03e3b30350a3d0b4"} Oct 30 21:42:35.862489 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:35.862463 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" event={"ID":"65c914e8-45f5-4219-9790-b5fc849d39f7","Type":"ContainerStarted","Data":"c8c79177b3da81ab600589898b3e7a11e1445286a2e2e82c991ea80d31d53b0b"} Oct 30 21:42:35.862584 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:35.862515 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jhjqr" event={"ID":"65c914e8-45f5-4219-9790-b5fc849d39f7","Type":"ContainerStarted","Data":"9d0aa3d71a2834f67a80cdd28d4a06e273193eee40fc6f337fb1df71bcfe0eb1"} Oct 30 21:42:35.864819 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:35.864801 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 21:42:35.864902 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:35.864846 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" event={"ID":"f9d15d9a-71ac-4572-98ca-c5f5df251cb2","Type":"ContainerStarted","Data":"a9d0109596de887a52aec08a7e9bf20823961590a97510768b36e9a6fbccd426"} Oct 30 21:42:35.865114 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:35.865089 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:35.870909 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:35.870885 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthdr" Oct 30 21:42:36.870317 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:36.870260 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l8jvk" event={"ID":"12730f3a-4d15-47f1-af22-9f4e37eedac7","Type":"ContainerStarted","Data":"19eeba53f32f68cfab54f360b4fb3366227f4b92a1eda0c5226c1f057b49bfb3"} Oct 30 21:42:36.870886 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:36.870861 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:36.871005 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:36.870907 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l8jvk" event={"ID":"12730f3a-4d15-47f1-af22-9f4e37eedac7","Type":"ContainerStarted","Data":"44e665653278db43dfd4eb51d2e7c79f6676329391f4456dd6e3a40087f2ceae"} Oct 30 21:42:39.766668 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:39.766614 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:39.767194 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:39.766692 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:39.767194 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:39.766718 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:39.767194 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:39.766740 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:39.767402 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:39.767377 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:39.767468 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:39.767453 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56e7aa71-e129-4ceb-91a9-06770acc40f7-service-ca-bundle\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:39.769000 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:39.768975 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c6a540-c40a-4a1e-b7c6-70e4384f1f24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r246j\" (UID: \"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24\") " pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:39.769150 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:39.769129 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e7aa71-e129-4ceb-91a9-06770acc40f7-metrics-certs\") pod \"router-default-755f8d7949-984mn\" (UID: \"56e7aa71-e129-4ceb-91a9-06770acc40f7\") " pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:40.048625 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:40.048542 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:40.061708 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:40.061678 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r246j" Oct 30 21:42:40.467825 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:40.467792 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68687f4c6c-hd5kb"] Oct 30 21:42:40.552999 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:40.552866 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r246j"] Oct 30 21:42:40.555924 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:40.555896 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c6a540_c40a_4a1e_b7c6_70e4384f1f24.slice/crio-89a2aa88aca99783afce115a48c9515f070900c49d8446b1a8465e4798f8baf1 WatchSource:0}: Error finding container 89a2aa88aca99783afce115a48c9515f070900c49d8446b1a8465e4798f8baf1: Status 404 returned error can't find the container with id 89a2aa88aca99783afce115a48c9515f070900c49d8446b1a8465e4798f8baf1 Oct 30 21:42:40.572033 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:40.572000 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-755f8d7949-984mn"] Oct 30 21:42:40.591032 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:40.589235 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e7aa71_e129_4ceb_91a9_06770acc40f7.slice/crio-cf2991c97ac3f60c19746822e0ebb8b03e4342a8b6d83daf599624ad46293d7e WatchSource:0}: Error finding container cf2991c97ac3f60c19746822e0ebb8b03e4342a8b6d83daf599624ad46293d7e: Status 404 returned error can't find the container with id cf2991c97ac3f60c19746822e0ebb8b03e4342a8b6d83daf599624ad46293d7e Oct 30 21:42:40.885780 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:40.885747 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r246j" event={"ID":"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24","Type":"ContainerStarted","Data":"e6db7c16c0c812177a215d22cf3790f419f83984a381e52d932ea029323c12e8"} Oct 30 21:42:40.886239 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:40.885810 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r246j" event={"ID":"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24","Type":"ContainerStarted","Data":"89a2aa88aca99783afce115a48c9515f070900c49d8446b1a8465e4798f8baf1"} Oct 30 21:42:40.887243 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:40.887222 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-755f8d7949-984mn" event={"ID":"56e7aa71-e129-4ceb-91a9-06770acc40f7","Type":"ContainerStarted","Data":"3aa16cd82792326ff4a0d832e526b4a53e439dcb924ab0d85edc65e3ba108ff3"} Oct 30 21:42:40.887343 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:40.887255 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-755f8d7949-984mn" event={"ID":"56e7aa71-e129-4ceb-91a9-06770acc40f7","Type":"ContainerStarted","Data":"cf2991c97ac3f60c19746822e0ebb8b03e4342a8b6d83daf599624ad46293d7e"} Oct 30 21:42:41.049286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:41.049253 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:41.051984 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:41.051929 4859 patch_prober.go:29] interesting pod/router-default-755f8d7949-984mn container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Oct 30 21:42:41.051984 ip-10-0-141-104 kubenswrapper[4859]: [-]has-synced failed: reason withheld Oct 30 21:42:41.051984 ip-10-0-141-104 kubenswrapper[4859]: [+]process-running ok Oct 30 21:42:41.051984 ip-10-0-141-104 kubenswrapper[4859]: healthz check failed Oct 30 21:42:41.052184 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:41.052004 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-755f8d7949-984mn" podUID="56e7aa71-e129-4ceb-91a9-06770acc40f7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 30 21:42:42.049904 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:42.049872 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:42.896266 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:42.896231 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r246j" event={"ID":"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24","Type":"ContainerStarted","Data":"dc9c66869706650886707713c02ac1445b1c500029538ff26180bb7afff37078"} Oct 30 21:42:42.896562 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:42.896545 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:42.897646 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:42.897628 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-755f8d7949-984mn" Oct 30 21:42:43.103290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:43.103257 4859 patch_prober.go:29] interesting pod/image-registry-76cdd49c4b-bf2dt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 21:42:43.103632 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:43.103312 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" podUID="8a474868-17e7-4ad9-a3f7-65376fc52940" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:42:44.611183 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:44.611148 4859 patch_prober.go:29] interesting pod/image-registry-76cdd49c4b-bf2dt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 21:42:44.611553 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:44.611214 4859 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" podUID="8a474868-17e7-4ad9-a3f7-65376fc52940" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:42:45.911352 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:45.911320 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r246j" event={"ID":"b2c6a540-c40a-4a1e-b7c6-70e4384f1f24","Type":"ContainerStarted","Data":"5d54abef4478db56a79e21881a0784d1915866b0c5c17f38521c03a2513a5382"} Oct 30 21:42:47.879591 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:47.879563 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l8jvk" Oct 30 21:42:50.473722 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:50.473696 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:42:51.358969 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.358920 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gczg9"] Oct 30 21:42:51.361674 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.361655 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.413424 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.413386 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:42:51.413424 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.413406 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:42:51.461863 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.461833 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-wtmp\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.462004 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.461871 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-tls\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.462004 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.461895 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.462076 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.462052 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c573208-a84b-4845-bbea-640f0ee93603-metrics-client-ca\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.462124 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.462108 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-accelerators-collector-config\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.462162 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.462139 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2c573208-a84b-4845-bbea-640f0ee93603-root\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.462162 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.462157 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-textfile\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.462223 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.462178 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtt55\" (UniqueName: \"kubernetes.io/projected/2c573208-a84b-4845-bbea-640f0ee93603-kube-api-access-vtt55\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.462255 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.462232 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c573208-a84b-4845-bbea-640f0ee93603-sys\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.462775 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.462753 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:42:51.462901 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.462780 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-csjdz\"" Oct 30 21:42:51.462901 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.462881 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:42:51.563679 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563649 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-wtmp\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.563679 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563684 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-tls\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563707 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563762 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c573208-a84b-4845-bbea-640f0ee93603-metrics-client-ca\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563789 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-accelerators-collector-config\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563821 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2c573208-a84b-4845-bbea-640f0ee93603-root\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563839 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-wtmp\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563848 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-textfile\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563902 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtt55\" (UniqueName: \"kubernetes.io/projected/2c573208-a84b-4845-bbea-640f0ee93603-kube-api-access-vtt55\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563911 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2c573208-a84b-4845-bbea-640f0ee93603-root\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563926 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c573208-a84b-4845-bbea-640f0ee93603-sys\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.563978 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c573208-a84b-4845-bbea-640f0ee93603-sys\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564741 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.564386 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-textfile\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564741 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.564559 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-accelerators-collector-config\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.564741 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.564564 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c573208-a84b-4845-bbea-640f0ee93603-metrics-client-ca\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.566192 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.566172 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.566285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.566218 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2c573208-a84b-4845-bbea-640f0ee93603-node-exporter-tls\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.620519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.620456 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtt55\" (UniqueName: \"kubernetes.io/projected/2c573208-a84b-4845-bbea-640f0ee93603-kube-api-access-vtt55\") pod \"node-exporter-gczg9\" (UID: \"2c573208-a84b-4845-bbea-640f0ee93603\") " pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.673462 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.673431 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gczg9" Oct 30 21:42:51.684037 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:42:51.684013 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c573208_a84b_4845_bbea_640f0ee93603.slice/crio-76172a8ea51ce7223cb394d8d3523524a8db713ccca52dbe515a649984503561 WatchSource:0}: Error finding container 76172a8ea51ce7223cb394d8d3523524a8db713ccca52dbe515a649984503561: Status 404 returned error can't find the container with id 76172a8ea51ce7223cb394d8d3523524a8db713ccca52dbe515a649984503561 Oct 30 21:42:51.931859 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:51.931778 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gczg9" event={"ID":"2c573208-a84b-4845-bbea-640f0ee93603","Type":"ContainerStarted","Data":"76172a8ea51ce7223cb394d8d3523524a8db713ccca52dbe515a649984503561"} Oct 30 21:42:53.103549 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:53.103524 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:42:53.155311 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:53.155279 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jnj6f" Oct 30 21:42:53.943803 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:53.943764 4859 generic.go:356] "Generic (PLEG): container finished" podID="2c573208-a84b-4845-bbea-640f0ee93603" containerID="12da90739c9bda3ce55daae0cd83b3d7f2e80377158531b6690c90eaa8391429" exitCode=0 Oct 30 21:42:53.944019 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:53.943841 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gczg9" event={"ID":"2c573208-a84b-4845-bbea-640f0ee93603","Type":"ContainerDied","Data":"12da90739c9bda3ce55daae0cd83b3d7f2e80377158531b6690c90eaa8391429"} Oct 30 21:42:54.949520 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:54.949484 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gczg9" event={"ID":"2c573208-a84b-4845-bbea-640f0ee93603","Type":"ContainerStarted","Data":"b5e7ed963f2e96be08e05b089565a4060ca0bfa5c5571b81f836bcfdcb8186d6"} Oct 30 21:42:54.949895 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:54.949534 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gczg9" event={"ID":"2c573208-a84b-4845-bbea-640f0ee93603","Type":"ContainerStarted","Data":"84efcf1418960b88be935865c0b7d861d37c59489f458951942f43bc080e0c78"} Oct 30 21:42:55.069089 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:42:55.069017 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gczg9" podStartSLOduration=2.649587338 podStartE2EDuration="4.069003422s" totalImagesPullingTime="1.352348753s" totalInitContainerRuntime="67.067331ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:55.067601314 +0000 UTC m=+32.440228061" watchObservedRunningTime="2026-10-30 21:42:55.069003422 +0000 UTC m=+32.441630180" Oct 30 21:43:03.267953 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:03.267908 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76cdd49c4b-bf2dt"] Oct 30 21:43:05.487608 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.487518 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" podUID="699c93ce-4424-4460-8cb2-b5db6ca24033" containerName="registry" containerID="cri-o://613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7" gracePeriod=30 Oct 30 21:43:05.721897 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.721869 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:43:05.784725 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.784669 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-certificates\") pod \"699c93ce-4424-4460-8cb2-b5db6ca24033\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " Oct 30 21:43:05.784725 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.784710 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") pod \"699c93ce-4424-4460-8cb2-b5db6ca24033\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " Oct 30 21:43:05.784862 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.784732 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-installation-pull-secrets\") pod \"699c93ce-4424-4460-8cb2-b5db6ca24033\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " Oct 30 21:43:05.784862 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.784754 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-image-registry-private-configuration\") pod \"699c93ce-4424-4460-8cb2-b5db6ca24033\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " Oct 30 21:43:05.784862 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.784816 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699c93ce-4424-4460-8cb2-b5db6ca24033-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/699c93ce-4424-4460-8cb2-b5db6ca24033-ca-trust-extracted\") pod \"699c93ce-4424-4460-8cb2-b5db6ca24033\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " Oct 30 21:43:05.785007 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.784869 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-kube-api-access-t7l4k\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-kube-api-access-t7l4k\") pod \"699c93ce-4424-4460-8cb2-b5db6ca24033\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " Oct 30 21:43:05.785007 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.784898 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-bound-sa-token\") pod \"699c93ce-4424-4460-8cb2-b5db6ca24033\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " Oct 30 21:43:05.785007 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.784923 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-trusted-ca\") pod \"699c93ce-4424-4460-8cb2-b5db6ca24033\" (UID: \"699c93ce-4424-4460-8cb2-b5db6ca24033\") " Oct 30 21:43:05.785440 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.785142 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-certificates" pod "699c93ce-4424-4460-8cb2-b5db6ca24033" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:05.785851 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.785812 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-trusted-ca" pod "699c93ce-4424-4460-8cb2-b5db6ca24033" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:05.787250 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.787220 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls" pod "699c93ce-4424-4460-8cb2-b5db6ca24033" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:05.787565 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.787535 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-kube-api-access-t7l4k" pod "699c93ce-4424-4460-8cb2-b5db6ca24033" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033"). InnerVolumeSpecName "kube-api-access-t7l4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:05.787642 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.787567 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-installation-pull-secrets" pod "699c93ce-4424-4460-8cb2-b5db6ca24033" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:05.787642 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.787630 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-image-registry-private-configuration" pod "699c93ce-4424-4460-8cb2-b5db6ca24033" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:05.787711 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.787661 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-bound-sa-token" pod "699c93ce-4424-4460-8cb2-b5db6ca24033" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:05.793903 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.793874 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699c93ce-4424-4460-8cb2-b5db6ca24033-ca-trust-extracted" pod "699c93ce-4424-4460-8cb2-b5db6ca24033" (UID: "699c93ce-4424-4460-8cb2-b5db6ca24033"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:05.885718 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.885690 4859 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/699c93ce-4424-4460-8cb2-b5db6ca24033-ca-trust-extracted\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:05.885718 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.885715 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7l4k\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-kube-api-access-t7l4k\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:05.885877 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.885727 4859 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-bound-sa-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:05.885877 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.885736 4859 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-trusted-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:05.885877 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.885745 4859 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-certificates\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:05.885877 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.885753 4859 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/699c93ce-4424-4460-8cb2-b5db6ca24033-registry-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:05.885877 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.885762 4859 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-installation-pull-secrets\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:05.885877 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.885771 4859 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/699c93ce-4424-4460-8cb2-b5db6ca24033-image-registry-private-configuration\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:05.991008 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.990973 4859 generic.go:356] "Generic (PLEG): container finished" podID="699c93ce-4424-4460-8cb2-b5db6ca24033" containerID="613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7" exitCode=0 Oct 30 21:43:05.991151 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.991051 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" event={"ID":"699c93ce-4424-4460-8cb2-b5db6ca24033","Type":"ContainerDied","Data":"613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7"} Oct 30 21:43:05.991151 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.991090 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" Oct 30 21:43:05.991151 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.991122 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68687f4c6c-hd5kb" event={"ID":"699c93ce-4424-4460-8cb2-b5db6ca24033","Type":"ContainerDied","Data":"283f0315a90b3bb0b2ff29e2ba46bd636db7e1c8cbc8b63564c1495080430dcf"} Oct 30 21:43:05.991151 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.991150 4859 scope.go:122] "RemoveContainer" containerID="613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7" Oct 30 21:43:05.999696 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:05.999677 4859 scope.go:122] "RemoveContainer" containerID="613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7" Oct 30 21:43:05.999999 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:43:05.999980 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7\": container with ID starting with 613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7 not found: ID does not exist" containerID="613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7" Oct 30 21:43:06.000054 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:06.000008 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7"} err="failed to get container status \"613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7\": rpc error: code = NotFound desc = could not find container \"613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7\": container with ID starting with 613cabcdf7e882cde0f7d60fc928185bb6a6fc11dcb43302de292886d93e59b7 not found: ID does not exist" Oct 30 21:43:06.153953 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:06.153911 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68687f4c6c-hd5kb"] Oct 30 21:43:06.158216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:06.158187 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68687f4c6c-hd5kb"] Oct 30 21:43:06.738940 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:06.738905 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699c93ce-4424-4460-8cb2-b5db6ca24033" path="/var/lib/kubelet/pods/699c93ce-4424-4460-8cb2-b5db6ca24033/volumes" Oct 30 21:43:09.875298 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:43:09.875265 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:43:28.286972 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.286903 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" podUID="8a474868-17e7-4ad9-a3f7-65376fc52940" containerName="registry" containerID="cri-o://d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1" gracePeriod=30 Oct 30 21:43:28.537481 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.537430 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:43:28.654354 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.654331 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-image-registry-private-configuration\") pod \"8a474868-17e7-4ad9-a3f7-65376fc52940\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " Oct 30 21:43:28.654482 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.654363 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-trusted-ca\") pod \"8a474868-17e7-4ad9-a3f7-65376fc52940\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " Oct 30 21:43:28.654482 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.654383 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-installation-pull-secrets\") pod \"8a474868-17e7-4ad9-a3f7-65376fc52940\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " Oct 30 21:43:28.654482 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.654419 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") pod \"8a474868-17e7-4ad9-a3f7-65376fc52940\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " Oct 30 21:43:28.654482 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.654452 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-kube-api-access-58245\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-kube-api-access-58245\") pod \"8a474868-17e7-4ad9-a3f7-65376fc52940\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " Oct 30 21:43:28.654698 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.654500 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-bound-sa-token\") pod \"8a474868-17e7-4ad9-a3f7-65376fc52940\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " Oct 30 21:43:28.654698 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.654537 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a474868-17e7-4ad9-a3f7-65376fc52940-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a474868-17e7-4ad9-a3f7-65376fc52940-ca-trust-extracted\") pod \"8a474868-17e7-4ad9-a3f7-65376fc52940\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " Oct 30 21:43:28.654698 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.654573 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-certificates\") pod \"8a474868-17e7-4ad9-a3f7-65376fc52940\" (UID: \"8a474868-17e7-4ad9-a3f7-65376fc52940\") " Oct 30 21:43:28.654863 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.654841 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-trusted-ca" pod "8a474868-17e7-4ad9-a3f7-65376fc52940" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:28.655146 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.655120 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-certificates" pod "8a474868-17e7-4ad9-a3f7-65376fc52940" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:28.656680 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.656645 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls" pod "8a474868-17e7-4ad9-a3f7-65376fc52940" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:28.656908 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.656861 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-installation-pull-secrets" pod "8a474868-17e7-4ad9-a3f7-65376fc52940" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:28.657061 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.657041 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-image-registry-private-configuration" pod "8a474868-17e7-4ad9-a3f7-65376fc52940" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:28.657233 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.657209 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-kube-api-access-58245" pod "8a474868-17e7-4ad9-a3f7-65376fc52940" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940"). InnerVolumeSpecName "kube-api-access-58245". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:28.657308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.657272 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-bound-sa-token" pod "8a474868-17e7-4ad9-a3f7-65376fc52940" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:28.661535 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.661504 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a474868-17e7-4ad9-a3f7-65376fc52940-ca-trust-extracted" pod "8a474868-17e7-4ad9-a3f7-65376fc52940" (UID: "8a474868-17e7-4ad9-a3f7-65376fc52940"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:28.755415 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.755381 4859 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-bound-sa-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:28.755415 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.755415 4859 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a474868-17e7-4ad9-a3f7-65376fc52940-ca-trust-extracted\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:28.755544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.755426 4859 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-certificates\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:28.755544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.755438 4859 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-image-registry-private-configuration\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:28.755544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.755447 4859 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a474868-17e7-4ad9-a3f7-65376fc52940-trusted-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:28.755544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.755456 4859 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a474868-17e7-4ad9-a3f7-65376fc52940-installation-pull-secrets\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:28.755544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.755464 4859 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-registry-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:28.755544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:28.755472 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58245\" (UniqueName: \"kubernetes.io/projected/8a474868-17e7-4ad9-a3f7-65376fc52940-kube-api-access-58245\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:29.071252 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:29.071215 4859 generic.go:356] "Generic (PLEG): container finished" podID="8a474868-17e7-4ad9-a3f7-65376fc52940" containerID="d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1" exitCode=0 Oct 30 21:43:29.071436 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:29.071264 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" Oct 30 21:43:29.071436 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:29.071264 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" event={"ID":"8a474868-17e7-4ad9-a3f7-65376fc52940","Type":"ContainerDied","Data":"d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1"} Oct 30 21:43:29.071436 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:29.071386 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cdd49c4b-bf2dt" event={"ID":"8a474868-17e7-4ad9-a3f7-65376fc52940","Type":"ContainerDied","Data":"c1c1b1816d545ea0b2ad686f7b56bb59901c3c51b13e04cb8d16061f9ceec800"} Oct 30 21:43:29.071436 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:29.071416 4859 scope.go:122] "RemoveContainer" containerID="d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1" Oct 30 21:43:29.079351 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:29.079331 4859 scope.go:122] "RemoveContainer" containerID="d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1" Oct 30 21:43:29.079649 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:43:29.079603 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1\": container with ID starting with d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1 not found: ID does not exist" containerID="d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1" Oct 30 21:43:29.079705 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:29.079654 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1"} err="failed to get container status \"d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1\": rpc error: code = NotFound desc = could not find container \"d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1\": container with ID starting with d12a60ba5d069b57997bca94ddfc28a05367334a0a749a4e95a8ab596b6a39f1 not found: ID does not exist" Oct 30 21:43:29.234117 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:29.234082 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76cdd49c4b-bf2dt"] Oct 30 21:43:29.283618 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:29.283585 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76cdd49c4b-bf2dt"] Oct 30 21:43:30.738030 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:30.737992 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a474868-17e7-4ad9-a3f7-65376fc52940" path="/var/lib/kubelet/pods/8a474868-17e7-4ad9-a3f7-65376fc52940/volumes" Oct 30 21:43:39.108936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:39.108906 4859 generic.go:356] "Generic (PLEG): container finished" podID="81fb1745-4867-438d-b4e0-d44e743e5230" containerID="2dabd9fc5fbc8c6697d0cdca6e615a8052539d688b52b22d038125154d6789bd" exitCode=0 Oct 30 21:43:39.109290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:39.108972 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" event={"ID":"81fb1745-4867-438d-b4e0-d44e743e5230","Type":"ContainerDied","Data":"2dabd9fc5fbc8c6697d0cdca6e615a8052539d688b52b22d038125154d6789bd"} Oct 30 21:43:39.109290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:39.109244 4859 scope.go:122] "RemoveContainer" containerID="2dabd9fc5fbc8c6697d0cdca6e615a8052539d688b52b22d038125154d6789bd" Oct 30 21:43:39.875652 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:43:39.875618 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:43:40.114272 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:40.114239 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q4vks" event={"ID":"81fb1745-4867-438d-b4e0-d44e743e5230","Type":"ContainerStarted","Data":"b9b44e3cfed5a68f6d63f67ae72ccdd5dce91e91609d817b8746274bf14e0d14"} Oct 30 21:43:48.141843 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:48.141809 4859 generic.go:356] "Generic (PLEG): container finished" podID="b21892bd-c791-419a-a1d7-4bdc643b7f07" containerID="939774b2ed2eb204240d907f6fe10fd742dcc74882930720063c773ffe191678" exitCode=0 Oct 30 21:43:48.142397 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:48.141872 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wbpqf" event={"ID":"b21892bd-c791-419a-a1d7-4bdc643b7f07","Type":"ContainerDied","Data":"939774b2ed2eb204240d907f6fe10fd742dcc74882930720063c773ffe191678"} Oct 30 21:43:48.142397 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:48.142299 4859 scope.go:122] "RemoveContainer" containerID="939774b2ed2eb204240d907f6fe10fd742dcc74882930720063c773ffe191678" Oct 30 21:43:49.146549 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:49.146514 4859 generic.go:356] "Generic (PLEG): container finished" podID="3bcc5627-b74e-49ed-a82b-36c2920b9917" containerID="fe760cab44c7134140a0afbf88ad15c5abab6a5e91e4ad50a50af579a2cf4cdb" exitCode=0 Oct 30 21:43:49.147064 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:49.146585 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" event={"ID":"3bcc5627-b74e-49ed-a82b-36c2920b9917","Type":"ContainerDied","Data":"fe760cab44c7134140a0afbf88ad15c5abab6a5e91e4ad50a50af579a2cf4cdb"} Oct 30 21:43:49.147064 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:49.146914 4859 scope.go:122] "RemoveContainer" containerID="fe760cab44c7134140a0afbf88ad15c5abab6a5e91e4ad50a50af579a2cf4cdb" Oct 30 21:43:49.148597 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:49.148567 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wbpqf" event={"ID":"b21892bd-c791-419a-a1d7-4bdc643b7f07","Type":"ContainerStarted","Data":"4789ab752dd341ee2466b864ff6993b8fe6dac227eff961c412f6f53adc86ea1"} Oct 30 21:43:50.153263 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:43:50.153231 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k2gbw" event={"ID":"3bcc5627-b74e-49ed-a82b-36c2920b9917","Type":"ContainerStarted","Data":"0a3588a8dfa2f18134b26021a0512d0ac233819d16cbdc63760abffdca44250d"} Oct 30 21:44:09.875808 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:44:09.875770 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:44:39.874929 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:44:39.874851 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:45:09.874563 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:45:09.874525 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:45:39.875244 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:45:39.875207 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:45:59.833491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.833457 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg"] Oct 30 21:45:59.833968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.833749 4859 cpu_manager.go:414] "containerMap: removing container" podUID="699c93ce-4424-4460-8cb2-b5db6ca24033" containerName="registry" Oct 30 21:45:59.833968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.833763 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699c93ce-4424-4460-8cb2-b5db6ca24033" containerName="registry" Oct 30 21:45:59.833968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.833781 4859 cpu_manager.go:414] "containerMap: removing container" podUID="8a474868-17e7-4ad9-a3f7-65376fc52940" containerName="registry" Oct 30 21:45:59.833968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.833787 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a474868-17e7-4ad9-a3f7-65376fc52940" containerName="registry" Oct 30 21:45:59.833968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.833834 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="699c93ce-4424-4460-8cb2-b5db6ca24033" containerName="registry" Oct 30 21:45:59.833968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.833847 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a474868-17e7-4ad9-a3f7-65376fc52940" containerName="registry" Oct 30 21:45:59.837042 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.836997 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:45:59.938518 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.938483 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:45:59.938518 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.938485 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:45:59.938786 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.938506 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-drrr5\"" Oct 30 21:45:59.939668 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.939646 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg"] Oct 30 21:45:59.940383 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.940364 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw"] Oct 30 21:45:59.943592 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:45:59.943574 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.007689 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.007658 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:00.007689 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.007692 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88xzf\" (UniqueName: \"kubernetes.io/projected/31448fe0-7459-49b7-a40f-7ee9376c57f3-kube-api-access-88xzf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:00.007968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.007834 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:00.008736 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.008717 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw"] Oct 30 21:46:00.019523 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.019502 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-zvknr\"" Oct 30 21:46:00.109087 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.109053 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:00.109279 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.109096 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74rs\" (UniqueName: \"kubernetes.io/projected/e7997836-978a-4993-af32-0631995a3099-kube-api-access-l74rs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.109279 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.109119 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.109279 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.109190 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.109279 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.109274 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:00.109490 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.109301 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88xzf\" (UniqueName: \"kubernetes.io/projected/31448fe0-7459-49b7-a40f-7ee9376c57f3-kube-api-access-88xzf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:00.109548 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.109500 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:00.109648 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.109625 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:00.168448 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.168411 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88xzf\" (UniqueName: \"kubernetes.io/projected/31448fe0-7459-49b7-a40f-7ee9376c57f3-kube-api-access-88xzf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:00.210146 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.210111 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74rs\" (UniqueName: \"kubernetes.io/projected/e7997836-978a-4993-af32-0631995a3099-kube-api-access-l74rs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.210146 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.210153 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.210391 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.210182 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.210593 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.210570 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.210649 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.210582 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.267734 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.267693 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74rs\" (UniqueName: \"kubernetes.io/projected/e7997836-978a-4993-af32-0631995a3099-kube-api-access-l74rs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.451738 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.451659 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:00.558977 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.558927 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:00.820493 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.820421 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg"] Oct 30 21:46:00.823456 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:00.823431 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31448fe0_7459_49b7_a40f_7ee9376c57f3.slice/crio-5608df9d0355c457e1539646b344701831386fc2f603f44ca9b8a080f5a4d371 WatchSource:0}: Error finding container 5608df9d0355c457e1539646b344701831386fc2f603f44ca9b8a080f5a4d371: Status 404 returned error can't find the container with id 5608df9d0355c457e1539646b344701831386fc2f603f44ca9b8a080f5a4d371 Oct 30 21:46:00.936123 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:00.936092 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw"] Oct 30 21:46:00.939464 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:00.939436 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7997836_978a_4993_af32_0631995a3099.slice/crio-c25e47f12aed0d76ee6dc6c936414a801d055bbe64511aa2117984854c2ac351 WatchSource:0}: Error finding container c25e47f12aed0d76ee6dc6c936414a801d055bbe64511aa2117984854c2ac351: Status 404 returned error can't find the container with id c25e47f12aed0d76ee6dc6c936414a801d055bbe64511aa2117984854c2ac351 Oct 30 21:46:01.595698 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:01.595646 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" event={"ID":"e7997836-978a-4993-af32-0631995a3099","Type":"ContainerStarted","Data":"c25e47f12aed0d76ee6dc6c936414a801d055bbe64511aa2117984854c2ac351"} Oct 30 21:46:01.596648 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:01.596626 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" event={"ID":"31448fe0-7459-49b7-a40f-7ee9376c57f3","Type":"ContainerStarted","Data":"5608df9d0355c457e1539646b344701831386fc2f603f44ca9b8a080f5a4d371"} Oct 30 21:46:03.082782 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.082744 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz"] Oct 30 21:46:03.086909 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.086491 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.189237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.187782 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ckv5h\"" Oct 30 21:46:03.189237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.189141 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz"] Oct 30 21:46:03.237000 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.236938 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wm4d\" (UniqueName: \"kubernetes.io/projected/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-kube-api-access-9wm4d\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.237165 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.237019 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.237165 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.237068 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.337701 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.337665 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.337874 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.337738 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.337874 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.337800 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wm4d\" (UniqueName: \"kubernetes.io/projected/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-kube-api-access-9wm4d\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.338740 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.338468 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.338740 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.338706 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.371256 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.371230 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66"] Oct 30 21:46:03.374808 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.374789 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:03.394686 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.394659 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wm4d\" (UniqueName: \"kubernetes.io/projected/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-kube-api-access-9wm4d\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.405360 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.405339 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:03.427916 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.427889 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-cf7dk\"" Oct 30 21:46:03.437068 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.437043 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66"] Oct 30 21:46:03.466870 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.466838 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p"] Oct 30 21:46:03.471161 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.470925 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:03.527644 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.527609 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-q7447\"" Oct 30 21:46:03.528919 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.528875 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p"] Oct 30 21:46:03.540072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.539807 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwb5k\" (UniqueName: \"kubernetes.io/projected/0e8bad54-57f5-47a2-837a-07e8bf95d745-kube-api-access-kwb5k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:03.540072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.539873 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:03.540072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.539982 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:03.568974 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.568929 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24"] Oct 30 21:46:03.573002 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.572981 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:03.627765 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.627684 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-xhxtv\"" Oct 30 21:46:03.630396 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.630367 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24"] Oct 30 21:46:03.641372 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.641337 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248jk\" (UniqueName: \"kubernetes.io/projected/81f16422-9fd7-4e74-bb28-9410321bc55b-kube-api-access-248jk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:03.641528 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.641459 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:03.641528 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.641500 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwb5k\" (UniqueName: \"kubernetes.io/projected/0e8bad54-57f5-47a2-837a-07e8bf95d745-kube-api-access-kwb5k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:03.641528 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.641525 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:03.641690 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.641562 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:03.641690 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.641588 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:03.642059 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.642038 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:03.642158 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.642040 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:03.665570 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.665536 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh"] Oct 30 21:46:03.669528 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.669501 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:03.701814 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.701781 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwb5k\" (UniqueName: \"kubernetes.io/projected/0e8bad54-57f5-47a2-837a-07e8bf95d745-kube-api-access-kwb5k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:03.726889 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.726851 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh"] Oct 30 21:46:03.727799 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.727771 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-smps4\"" Oct 30 21:46:03.742615 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.742580 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248jk\" (UniqueName: \"kubernetes.io/projected/81f16422-9fd7-4e74-bb28-9410321bc55b-kube-api-access-248jk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:03.742785 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.742703 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:03.742863 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.742743 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:03.742863 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.742836 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:03.743007 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.742982 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw452\" (UniqueName: \"kubernetes.io/projected/aea0bddf-6408-40e4-927a-9dfec759d091-kube-api-access-zw452\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:03.743067 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.743015 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:03.743155 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.743133 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:03.743342 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.743292 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:03.788545 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.788508 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz"] Oct 30 21:46:03.792396 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:03.792355 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d1e6c9_dadc_4306_b1cc_aa442d4fa8f3.slice/crio-e3a30a86ed89852e64482c1902f4eef915d782e93d224fb719b7095b4af18882 WatchSource:0}: Error finding container e3a30a86ed89852e64482c1902f4eef915d782e93d224fb719b7095b4af18882: Status 404 returned error can't find the container with id e3a30a86ed89852e64482c1902f4eef915d782e93d224fb719b7095b4af18882 Oct 30 21:46:03.802460 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.802426 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248jk\" (UniqueName: \"kubernetes.io/projected/81f16422-9fd7-4e74-bb28-9410321bc55b-kube-api-access-248jk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:03.843576 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.843539 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw452\" (UniqueName: \"kubernetes.io/projected/aea0bddf-6408-40e4-927a-9dfec759d091-kube-api-access-zw452\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:03.843735 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.843598 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:03.843735 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.843646 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:03.843735 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.843678 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:03.843735 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.843707 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl8xq\" (UniqueName: \"kubernetes.io/projected/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-kube-api-access-hl8xq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:03.843912 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.843749 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:03.844231 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.844203 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:03.844762 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.844739 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:03.901689 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.901612 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw452\" (UniqueName: \"kubernetes.io/projected/aea0bddf-6408-40e4-927a-9dfec759d091-kube-api-access-zw452\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:03.945034 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.944985 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:03.945034 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.945038 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:03.945245 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.945068 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl8xq\" (UniqueName: \"kubernetes.io/projected/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-kube-api-access-hl8xq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:03.945418 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.945398 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:03.945475 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.945435 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:03.993024 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:03.992979 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:04.002852 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.002829 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl8xq\" (UniqueName: \"kubernetes.io/projected/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-kube-api-access-hl8xq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:04.089489 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.089454 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:04.191548 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.191517 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:04.287814 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.287550 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:04.379806 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.379721 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66"] Oct 30 21:46:04.385989 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:04.385923 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8bad54_57f5_47a2_837a_07e8bf95d745.slice/crio-1d6932f7f263b650559336d2e51072eae82482bd31435d854e36aecc32cc0fa3 WatchSource:0}: Error finding container 1d6932f7f263b650559336d2e51072eae82482bd31435d854e36aecc32cc0fa3: Status 404 returned error can't find the container with id 1d6932f7f263b650559336d2e51072eae82482bd31435d854e36aecc32cc0fa3 Oct 30 21:46:04.471873 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.471841 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p"] Oct 30 21:46:04.474263 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:04.474229 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f16422_9fd7_4e74_bb28_9410321bc55b.slice/crio-f6e1d0a59e6477f60bc3d2cebc90f0668cb03317a3e2586f3d176df3e44d3411 WatchSource:0}: Error finding container f6e1d0a59e6477f60bc3d2cebc90f0668cb03317a3e2586f3d176df3e44d3411: Status 404 returned error can't find the container with id f6e1d0a59e6477f60bc3d2cebc90f0668cb03317a3e2586f3d176df3e44d3411 Oct 30 21:46:04.572748 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.572577 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24"] Oct 30 21:46:04.575697 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:04.575650 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea0bddf_6408_40e4_927a_9dfec759d091.slice/crio-a02b4024fa50bcc379bcb93b354511d14a40abbcaaa6eeb49bd2ac70bfd2b634 WatchSource:0}: Error finding container a02b4024fa50bcc379bcb93b354511d14a40abbcaaa6eeb49bd2ac70bfd2b634: Status 404 returned error can't find the container with id a02b4024fa50bcc379bcb93b354511d14a40abbcaaa6eeb49bd2ac70bfd2b634 Oct 30 21:46:04.611446 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.611411 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" event={"ID":"aea0bddf-6408-40e4-927a-9dfec759d091","Type":"ContainerStarted","Data":"a02b4024fa50bcc379bcb93b354511d14a40abbcaaa6eeb49bd2ac70bfd2b634"} Oct 30 21:46:04.612878 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.612849 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" event={"ID":"81f16422-9fd7-4e74-bb28-9410321bc55b","Type":"ContainerStarted","Data":"f6e1d0a59e6477f60bc3d2cebc90f0668cb03317a3e2586f3d176df3e44d3411"} Oct 30 21:46:04.614774 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.614749 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" event={"ID":"0e8bad54-57f5-47a2-837a-07e8bf95d745","Type":"ContainerStarted","Data":"1d6932f7f263b650559336d2e51072eae82482bd31435d854e36aecc32cc0fa3"} Oct 30 21:46:04.616170 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.616145 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" event={"ID":"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3","Type":"ContainerStarted","Data":"e3a30a86ed89852e64482c1902f4eef915d782e93d224fb719b7095b4af18882"} Oct 30 21:46:04.674911 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:04.674871 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh"] Oct 30 21:46:04.677880 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:04.677849 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4357ed8_2175_40dc_b2bf_97617c6a6ad9.slice/crio-54598e72c7e378f6ab490e453614c24c26019a0abca00a6883fe5f9fa2c24a0b WatchSource:0}: Error finding container 54598e72c7e378f6ab490e453614c24c26019a0abca00a6883fe5f9fa2c24a0b: Status 404 returned error can't find the container with id 54598e72c7e378f6ab490e453614c24c26019a0abca00a6883fe5f9fa2c24a0b Oct 30 21:46:05.623633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:05.623394 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" event={"ID":"a4357ed8-2175-40dc-b2bf-97617c6a6ad9","Type":"ContainerStarted","Data":"54598e72c7e378f6ab490e453614c24c26019a0abca00a6883fe5f9fa2c24a0b"} Oct 30 21:46:06.376321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.374568 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm"] Oct 30 21:46:06.379154 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.378553 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.425606 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.423671 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f"] Oct 30 21:46:06.427727 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.427696 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:06.430616 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.430578 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:46:06.430799 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.430771 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dz4rf\"" Oct 30 21:46:06.477775 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.477740 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.477939 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.477788 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.477939 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.477892 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.478080 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.477957 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wql7j\" (UniqueName: \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-kube-api-access-wql7j\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.478080 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.478052 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.478190 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.478086 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.478190 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.478115 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.478300 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.478283 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.478356 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.478332 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.481901 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.481538 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f"] Oct 30 21:46:06.484774 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.484711 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm"] Oct 30 21:46:06.579058 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579009 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wql7j\" (UniqueName: \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-kube-api-access-wql7j\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.579216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579067 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.579216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579100 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.579216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579125 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.579216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579182 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:06.579440 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579218 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.579440 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579257 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjx95\" (UniqueName: \"kubernetes.io/projected/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-kube-api-access-cjx95\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:06.579440 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579288 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.579440 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579328 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.579440 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579361 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.579440 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579396 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.579440 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.579423 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:06.581236 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.580193 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.581236 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.580211 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.581236 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.580857 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.581236 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.580857 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.581236 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.580938 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.593294 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.593251 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.598756 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.598706 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.638404 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.638306 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wql7j\" (UniqueName: \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-kube-api-access-wql7j\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.643835 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.643800 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-xxpxm\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.680275 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.680198 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:06.680432 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.680320 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjx95\" (UniqueName: \"kubernetes.io/projected/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-kube-api-access-cjx95\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:06.680432 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.680386 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:06.680833 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.680805 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:06.681540 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.681517 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:06.689441 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.687335 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm"] Oct 30 21:46:06.692644 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.692577 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.703626 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.703352 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:06.750296 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.749753 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjx95\" (UniqueName: \"kubernetes.io/projected/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-kube-api-access-cjx95\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:06.759094 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.759028 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-6rckl\"" Oct 30 21:46:06.762078 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.761985 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm"] Oct 30 21:46:06.882213 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.882184 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zn9f\" (UniqueName: \"kubernetes.io/projected/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-kube-api-access-2zn9f\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.882405 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.882236 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.882405 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.882258 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.882405 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.882282 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.882405 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.882302 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.882405 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.882329 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.882405 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.882343 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.882405 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.882388 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.882760 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.882420 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.983337 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.983259 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.983524 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.983339 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zn9f\" (UniqueName: \"kubernetes.io/projected/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-kube-api-access-2zn9f\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.983524 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.983382 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.983524 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.983416 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.983524 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.983452 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.983524 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.983482 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.983719 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.983533 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.983719 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.983558 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.983719 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.983609 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.984468 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.984437 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.985232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.985189 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.985933 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.985911 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.986195 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.986173 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.986720 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.986698 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.987811 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.987766 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:06.988546 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:06.988525 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:07.045039 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.044965 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:07.047880 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.047692 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:07.051288 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.051264 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zn9f\" (UniqueName: \"kubernetes.io/projected/3306eb24-d85b-4e7e-b4b4-e4f71c01050b-kube-api-access-2zn9f\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm\" (UID: \"3306eb24-d85b-4e7e-b4b4-e4f71c01050b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:07.167219 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.167182 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb"] Oct 30 21:46:07.172561 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.172535 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.196306 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.196203 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm"] Oct 30 21:46:07.202353 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:07.202321 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcf5ae0_b25d_42d0_b0b3_1fedebf3a103.slice/crio-9fbb34fa58270baccd210a129efc3f8b820979260decc95ed054c05260e8c38f WatchSource:0}: Error finding container 9fbb34fa58270baccd210a129efc3f8b820979260decc95ed054c05260e8c38f: Status 404 returned error can't find the container with id 9fbb34fa58270baccd210a129efc3f8b820979260decc95ed054c05260e8c38f Oct 30 21:46:07.229632 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.229606 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb"] Oct 30 21:46:07.286585 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.286318 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.286585 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.286401 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.286585 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.286443 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czwfh\" (UniqueName: \"kubernetes.io/projected/061abe43-b078-4bef-ab54-4a6dcbeb56fc-kube-api-access-czwfh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.313331 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.313291 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:07.387805 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.387762 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwfh\" (UniqueName: \"kubernetes.io/projected/061abe43-b078-4bef-ab54-4a6dcbeb56fc-kube-api-access-czwfh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.392850 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.387831 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.392850 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.387933 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.392850 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.388345 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.392850 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.388897 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.460487 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.460426 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwfh\" (UniqueName: \"kubernetes.io/projected/061abe43-b078-4bef-ab54-4a6dcbeb56fc-kube-api-access-czwfh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.496297 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.496226 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:07.523418 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.522515 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f"] Oct 30 21:46:07.641500 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.641460 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" event={"ID":"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae","Type":"ContainerStarted","Data":"9e4b31756ee3967b65a22b1f73f319bdd949390991c2187935dfe2f499227355"} Oct 30 21:46:07.643585 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.643552 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" event={"ID":"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103","Type":"ContainerStarted","Data":"9fbb34fa58270baccd210a129efc3f8b820979260decc95ed054c05260e8c38f"} Oct 30 21:46:07.732425 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.732392 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm"] Oct 30 21:46:07.735002 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:07.734969 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3306eb24_d85b_4e7e_b4b4_e4f71c01050b.slice/crio-68ce26c62f6149bfc4905bcfdfe8c7a6f959c3d7d2f05341c0e6a4e4411136b2 WatchSource:0}: Error finding container 68ce26c62f6149bfc4905bcfdfe8c7a6f959c3d7d2f05341c0e6a4e4411136b2: Status 404 returned error can't find the container with id 68ce26c62f6149bfc4905bcfdfe8c7a6f959c3d7d2f05341c0e6a4e4411136b2 Oct 30 21:46:07.884242 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:07.883362 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb"] Oct 30 21:46:07.887372 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:07.887318 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061abe43_b078_4bef_ab54_4a6dcbeb56fc.slice/crio-ec9d15cde200382d143d573eb28e36efe0439fd71e481b50887cc718b976dae8 WatchSource:0}: Error finding container ec9d15cde200382d143d573eb28e36efe0439fd71e481b50887cc718b976dae8: Status 404 returned error can't find the container with id ec9d15cde200382d143d573eb28e36efe0439fd71e481b50887cc718b976dae8 Oct 30 21:46:08.656694 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:08.656510 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" event={"ID":"3306eb24-d85b-4e7e-b4b4-e4f71c01050b","Type":"ContainerStarted","Data":"68ce26c62f6149bfc4905bcfdfe8c7a6f959c3d7d2f05341c0e6a4e4411136b2"} Oct 30 21:46:08.659637 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:08.659569 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" event={"ID":"061abe43-b078-4bef-ab54-4a6dcbeb56fc","Type":"ContainerStarted","Data":"ec9d15cde200382d143d573eb28e36efe0439fd71e481b50887cc718b976dae8"} Oct 30 21:46:09.896659 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:46:09.896574 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:46:14.735981 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:14.733845 4859 generic.go:356] "Generic (PLEG): container finished" podID="31448fe0-7459-49b7-a40f-7ee9376c57f3" containerID="073690efa79cd31c5eade3446ed236ee50f0110f9b02e19583da08698558a38c" exitCode=0 Oct 30 21:46:14.744888 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:14.744817 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" event={"ID":"31448fe0-7459-49b7-a40f-7ee9376c57f3","Type":"ContainerDied","Data":"073690efa79cd31c5eade3446ed236ee50f0110f9b02e19583da08698558a38c"} Oct 30 21:46:15.743152 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:15.743116 4859 generic.go:356] "Generic (PLEG): container finished" podID="aea0bddf-6408-40e4-927a-9dfec759d091" containerID="07594a2fdd5a97d9eb16e889ae0f3abc13b580f983a0b33d52ad6627f46cfb97" exitCode=0 Oct 30 21:46:15.743562 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:15.743248 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" event={"ID":"aea0bddf-6408-40e4-927a-9dfec759d091","Type":"ContainerDied","Data":"07594a2fdd5a97d9eb16e889ae0f3abc13b580f983a0b33d52ad6627f46cfb97"} Oct 30 21:46:15.749359 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:15.749327 4859 generic.go:356] "Generic (PLEG): container finished" podID="81f16422-9fd7-4e74-bb28-9410321bc55b" containerID="28d9ffb02110ece2936bc2336756717a7e4faaa373d94231763da48aee75be66" exitCode=0 Oct 30 21:46:15.749467 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:15.749436 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" event={"ID":"81f16422-9fd7-4e74-bb28-9410321bc55b","Type":"ContainerDied","Data":"28d9ffb02110ece2936bc2336756717a7e4faaa373d94231763da48aee75be66"} Oct 30 21:46:15.759771 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:15.759742 4859 generic.go:356] "Generic (PLEG): container finished" podID="0e8bad54-57f5-47a2-837a-07e8bf95d745" containerID="c095b1267efbdd5bee2abb586c7afdf11a34f2a9aef426af7a9c35879f9ff615" exitCode=0 Oct 30 21:46:15.759883 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:15.759844 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" event={"ID":"0e8bad54-57f5-47a2-837a-07e8bf95d745","Type":"ContainerDied","Data":"c095b1267efbdd5bee2abb586c7afdf11a34f2a9aef426af7a9c35879f9ff615"} Oct 30 21:46:15.764400 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:15.763667 4859 generic.go:356] "Generic (PLEG): container finished" podID="e7997836-978a-4993-af32-0631995a3099" containerID="8c06e8fee9dca4d335a86bd9bc54d243817687709af08bd79bf4c7f0edb8428c" exitCode=0 Oct 30 21:46:15.764400 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:15.763748 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" event={"ID":"e7997836-978a-4993-af32-0631995a3099","Type":"ContainerDied","Data":"8c06e8fee9dca4d335a86bd9bc54d243817687709af08bd79bf4c7f0edb8428c"} Oct 30 21:46:16.547690 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:16.546966 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:16.547690 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:16.547059 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:16.547690 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:16.547109 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:16.776179 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:16.776140 4859 generic.go:356] "Generic (PLEG): container finished" podID="a4357ed8-2175-40dc-b2bf-97617c6a6ad9" containerID="5374ac8f47b38475c7b783785d97d303a611a212ba07cbc3cdc8f2a909b0e2fa" exitCode=0 Oct 30 21:46:16.776638 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:16.776190 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" event={"ID":"a4357ed8-2175-40dc-b2bf-97617c6a6ad9","Type":"ContainerDied","Data":"5374ac8f47b38475c7b783785d97d303a611a212ba07cbc3cdc8f2a909b0e2fa"} Oct 30 21:46:16.780193 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:16.780155 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" event={"ID":"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103","Type":"ContainerStarted","Data":"e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2"} Oct 30 21:46:16.967429 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:16.967385 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:16.967544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:16.967475 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:16.967544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:16.967515 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:17.704321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:17.704287 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:17.709843 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:17.709816 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:17.798753 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:17.798660 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" event={"ID":"3306eb24-d85b-4e7e-b4b4-e4f71c01050b","Type":"ContainerStarted","Data":"d22acab7405068cb1b29820f0f89486abdb84dda2685abf9bb601a688aeec324"} Oct 30 21:46:17.799163 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:17.799095 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:17.800039 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:17.800006 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:46:17.827225 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:17.827176 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" podStartSLOduration=2.50033715 podStartE2EDuration="11.827160753s" totalImagesPullingTime="9.326823603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:17.014245535 +0000 UTC m=+234.386872283" watchObservedRunningTime="2026-10-30 21:46:17.827160753 +0000 UTC m=+235.199787501" Oct 30 21:46:18.059409 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.059345 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" podStartSLOduration=2.83079091 podStartE2EDuration="12.059326625s" totalImagesPullingTime="9.228535715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:18.058123029 +0000 UTC m=+235.430749801" watchObservedRunningTime="2026-10-30 21:46:18.059326625 +0000 UTC m=+235.431953373" Oct 30 21:46:18.314505 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.314471 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:18.320077 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.320048 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:18.807325 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.807286 4859 generic.go:356] "Generic (PLEG): container finished" podID="a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" containerID="e0aaccaf6864e35828e5286919860f00eb47dbcaa7995d3613c0546c7a17c669" exitCode=0 Oct 30 21:46:18.807776 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.807380 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" event={"ID":"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae","Type":"ContainerDied","Data":"e0aaccaf6864e35828e5286919860f00eb47dbcaa7995d3613c0546c7a17c669"} Oct 30 21:46:18.810721 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.810690 4859 generic.go:356] "Generic (PLEG): container finished" podID="a4357ed8-2175-40dc-b2bf-97617c6a6ad9" containerID="3fd74576c9d2d55ef83406bd1e515a965d7033e34edbe8eddde37c047ababfd6" exitCode=0 Oct 30 21:46:18.810827 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.810733 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" event={"ID":"a4357ed8-2175-40dc-b2bf-97617c6a6ad9","Type":"ContainerDied","Data":"3fd74576c9d2d55ef83406bd1e515a965d7033e34edbe8eddde37c047ababfd6"} Oct 30 21:46:18.813923 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.813901 4859 generic.go:356] "Generic (PLEG): container finished" podID="061abe43-b078-4bef-ab54-4a6dcbeb56fc" containerID="b8727847755aa02008bf5c097a05fc6f01d3002da515201f25fecb512dbfeca0" exitCode=0 Oct 30 21:46:18.814598 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.814523 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" event={"ID":"061abe43-b078-4bef-ab54-4a6dcbeb56fc","Type":"ContainerDied","Data":"b8727847755aa02008bf5c097a05fc6f01d3002da515201f25fecb512dbfeca0"} Oct 30 21:46:18.818290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.818248 4859 generic.go:356] "Generic (PLEG): container finished" podID="aea0bddf-6408-40e4-927a-9dfec759d091" containerID="930f6f954660c85119aa2af3364b85738ef3b566904ef717ce9b06242706941d" exitCode=0 Oct 30 21:46:18.818384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.818364 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" event={"ID":"aea0bddf-6408-40e4-927a-9dfec759d091","Type":"ContainerDied","Data":"930f6f954660c85119aa2af3364b85738ef3b566904ef717ce9b06242706941d"} Oct 30 21:46:18.820669 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.820645 4859 generic.go:356] "Generic (PLEG): container finished" podID="81f16422-9fd7-4e74-bb28-9410321bc55b" containerID="67d448f7d5a62208ecb899fde0b0292055c1b42ae84ef1fa866ce9f7048226d6" exitCode=0 Oct 30 21:46:18.820761 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.820743 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" event={"ID":"81f16422-9fd7-4e74-bb28-9410321bc55b","Type":"ContainerDied","Data":"67d448f7d5a62208ecb899fde0b0292055c1b42ae84ef1fa866ce9f7048226d6"} Oct 30 21:46:18.823050 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.823017 4859 generic.go:356] "Generic (PLEG): container finished" podID="0e8bad54-57f5-47a2-837a-07e8bf95d745" containerID="a8ebdd1c879bb11c2616659c4a9ab92350fc54b088284761099b6a300ee4a264" exitCode=0 Oct 30 21:46:18.825514 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.824363 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" event={"ID":"0e8bad54-57f5-47a2-837a-07e8bf95d745","Type":"ContainerDied","Data":"a8ebdd1c879bb11c2616659c4a9ab92350fc54b088284761099b6a300ee4a264"} Oct 30 21:46:18.825514 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.824408 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:18.826007 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:18.825989 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm" Oct 30 21:46:19.832959 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:19.832906 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" event={"ID":"e7997836-978a-4993-af32-0631995a3099","Type":"ContainerStarted","Data":"ea3d461b7afd24a868ec817b7f80a82678f0b6debfbf27df3ca31378646c18d8"} Oct 30 21:46:19.836676 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:19.835917 4859 generic.go:356] "Generic (PLEG): container finished" podID="31448fe0-7459-49b7-a40f-7ee9376c57f3" containerID="cc7133305f2d1bd8e681af52101f0cb5abc98a122911e91087a3ee8fdc9d6309" exitCode=0 Oct 30 21:46:19.836676 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:19.836004 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" event={"ID":"31448fe0-7459-49b7-a40f-7ee9376c57f3","Type":"ContainerDied","Data":"cc7133305f2d1bd8e681af52101f0cb5abc98a122911e91087a3ee8fdc9d6309"} Oct 30 21:46:19.840192 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:19.840141 4859 generic.go:356] "Generic (PLEG): container finished" podID="63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" containerID="b3cf53815986e0c8e97ae9fbde0f150a7158ad2546ade2ad87454eee0a7c86c6" exitCode=0 Oct 30 21:46:19.841679 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:19.841656 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" event={"ID":"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3","Type":"ContainerDied","Data":"b3cf53815986e0c8e97ae9fbde0f150a7158ad2546ade2ad87454eee0a7c86c6"} Oct 30 21:46:20.848730 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:20.848681 4859 generic.go:356] "Generic (PLEG): container finished" podID="a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" containerID="873a7b838161354cd58d7c4d9293418497478fc616a6dc690d41ef4574d79abc" exitCode=0 Oct 30 21:46:20.849187 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:20.848772 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" event={"ID":"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae","Type":"ContainerDied","Data":"873a7b838161354cd58d7c4d9293418497478fc616a6dc690d41ef4574d79abc"} Oct 30 21:46:20.854130 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:20.853468 4859 generic.go:356] "Generic (PLEG): container finished" podID="e7997836-978a-4993-af32-0631995a3099" containerID="ea3d461b7afd24a868ec817b7f80a82678f0b6debfbf27df3ca31378646c18d8" exitCode=0 Oct 30 21:46:20.854130 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:20.853558 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" event={"ID":"e7997836-978a-4993-af32-0631995a3099","Type":"ContainerDied","Data":"ea3d461b7afd24a868ec817b7f80a82678f0b6debfbf27df3ca31378646c18d8"} Oct 30 21:46:20.858966 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:20.858575 4859 generic.go:356] "Generic (PLEG): container finished" podID="061abe43-b078-4bef-ab54-4a6dcbeb56fc" containerID="0050ddfd28e4be0298670ab238d778e7b6a97bc5d6552f5a98caf180d3d5f559" exitCode=0 Oct 30 21:46:20.858966 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:20.858711 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" event={"ID":"061abe43-b078-4bef-ab54-4a6dcbeb56fc","Type":"ContainerDied","Data":"0050ddfd28e4be0298670ab238d778e7b6a97bc5d6552f5a98caf180d3d5f559"} Oct 30 21:46:21.872782 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:21.872079 4859 generic.go:356] "Generic (PLEG): container finished" podID="63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" containerID="c155c54de6afab974f39ff851378ec272868f5478c1745de7ab4565bb2fafab8" exitCode=0 Oct 30 21:46:21.872782 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:21.872124 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" event={"ID":"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3","Type":"ContainerDied","Data":"c155c54de6afab974f39ff851378ec272868f5478c1745de7ab4565bb2fafab8"} Oct 30 21:46:37.065990 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:37.065474 4859 generic.go:356] "Generic (PLEG): container finished" podID="0e8bad54-57f5-47a2-837a-07e8bf95d745" containerID="70bbeb994d272fc40571cea6aea75ef1f13fbe9da8e34d3c641ac77cb4b4ca4d" exitCode=0 Oct 30 21:46:37.065990 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:37.065574 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" event={"ID":"0e8bad54-57f5-47a2-837a-07e8bf95d745","Type":"ContainerDied","Data":"70bbeb994d272fc40571cea6aea75ef1f13fbe9da8e34d3c641ac77cb4b4ca4d"} Oct 30 21:46:37.070532 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:37.070500 4859 generic.go:356] "Generic (PLEG): container finished" podID="a4357ed8-2175-40dc-b2bf-97617c6a6ad9" containerID="1526e15947e8e04b8260a80fe02313a8a5dde1300366f3d21a041c5cf2a0247a" exitCode=0 Oct 30 21:46:37.070668 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:37.070634 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" event={"ID":"a4357ed8-2175-40dc-b2bf-97617c6a6ad9","Type":"ContainerDied","Data":"1526e15947e8e04b8260a80fe02313a8a5dde1300366f3d21a041c5cf2a0247a"} Oct 30 21:46:37.075627 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:37.075596 4859 generic.go:356] "Generic (PLEG): container finished" podID="aea0bddf-6408-40e4-927a-9dfec759d091" containerID="d8c0267ca64992128e97fc61e79f299da22d0e056c01e48a083f35267390a85a" exitCode=0 Oct 30 21:46:37.075748 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:37.075727 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" event={"ID":"aea0bddf-6408-40e4-927a-9dfec759d091","Type":"ContainerDied","Data":"d8c0267ca64992128e97fc61e79f299da22d0e056c01e48a083f35267390a85a"} Oct 30 21:46:37.079798 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:37.079770 4859 generic.go:356] "Generic (PLEG): container finished" podID="81f16422-9fd7-4e74-bb28-9410321bc55b" containerID="32386c7f0018f178f53c8285a8f70f8df10cd6012a73fb1d239401de89ddb4dc" exitCode=0 Oct 30 21:46:37.079924 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:37.079822 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" event={"ID":"81f16422-9fd7-4e74-bb28-9410321bc55b","Type":"ContainerDied","Data":"32386c7f0018f178f53c8285a8f70f8df10cd6012a73fb1d239401de89ddb4dc"} Oct 30 21:46:38.091091 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.090353 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" event={"ID":"31448fe0-7459-49b7-a40f-7ee9376c57f3","Type":"ContainerStarted","Data":"5ef21e8c74b82ab822c8d974e18ec7ef358d0bb3696227e5586e909902859366"} Oct 30 21:46:38.219491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.219358 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" podStartSLOduration=3.768132602 podStartE2EDuration="39.21933545s" totalImagesPullingTime="35.451202848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:38.208591385 +0000 UTC m=+255.581218132" watchObservedRunningTime="2026-10-30 21:46:38.21933545 +0000 UTC m=+255.591962200" Oct 30 21:46:38.282142 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.270234 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:38.385187 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.378311 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:38.392770 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.392679 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:38.410016 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.409991 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:38.441276 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.441250 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-bundle\") pod \"0e8bad54-57f5-47a2-837a-07e8bf95d745\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " Oct 30 21:46:38.441404 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.441299 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8bad54-57f5-47a2-837a-07e8bf95d745-kube-api-access-kwb5k\" (UniqueName: \"kubernetes.io/projected/0e8bad54-57f5-47a2-837a-07e8bf95d745-kube-api-access-kwb5k\") pod \"0e8bad54-57f5-47a2-837a-07e8bf95d745\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " Oct 30 21:46:38.441462 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.441405 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-util\" (UniqueName: \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-util\") pod \"0e8bad54-57f5-47a2-837a-07e8bf95d745\" (UID: \"0e8bad54-57f5-47a2-837a-07e8bf95d745\") " Oct 30 21:46:38.442325 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.442281 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-bundle" pod "0e8bad54-57f5-47a2-837a-07e8bf95d745" (UID: "0e8bad54-57f5-47a2-837a-07e8bf95d745"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:38.446888 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.446865 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-util" pod "0e8bad54-57f5-47a2-837a-07e8bf95d745" (UID: "0e8bad54-57f5-47a2-837a-07e8bf95d745"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:38.453046 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.453019 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8bad54-57f5-47a2-837a-07e8bf95d745-kube-api-access-kwb5k" pod "0e8bad54-57f5-47a2-837a-07e8bf95d745" (UID: "0e8bad54-57f5-47a2-837a-07e8bf95d745"). InnerVolumeSpecName "kube-api-access-kwb5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542268 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea0bddf-6408-40e4-927a-9dfec759d091-kube-api-access-zw452\" (UniqueName: \"kubernetes.io/projected/aea0bddf-6408-40e4-927a-9dfec759d091-kube-api-access-zw452\") pod \"aea0bddf-6408-40e4-927a-9dfec759d091\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542330 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-kube-api-access-hl8xq\" (UniqueName: \"kubernetes.io/projected/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-kube-api-access-hl8xq\") pod \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542366 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-util\" (UniqueName: \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-util\") pod \"aea0bddf-6408-40e4-927a-9dfec759d091\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542392 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-util\" (UniqueName: \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-util\") pod \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542428 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-bundle\") pod \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\" (UID: \"a4357ed8-2175-40dc-b2bf-97617c6a6ad9\") " Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542469 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f16422-9fd7-4e74-bb28-9410321bc55b-kube-api-access-248jk\" (UniqueName: \"kubernetes.io/projected/81f16422-9fd7-4e74-bb28-9410321bc55b-kube-api-access-248jk\") pod \"81f16422-9fd7-4e74-bb28-9410321bc55b\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542516 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-bundle\") pod \"81f16422-9fd7-4e74-bb28-9410321bc55b\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542562 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-bundle\") pod \"aea0bddf-6408-40e4-927a-9dfec759d091\" (UID: \"aea0bddf-6408-40e4-927a-9dfec759d091\") " Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542610 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-util\" (UniqueName: \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-util\") pod \"81f16422-9fd7-4e74-bb28-9410321bc55b\" (UID: \"81f16422-9fd7-4e74-bb28-9410321bc55b\") " Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542831 4859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-util\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542847 4859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e8bad54-57f5-47a2-837a-07e8bf95d745-bundle\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.543072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.542861 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwb5k\" (UniqueName: \"kubernetes.io/projected/0e8bad54-57f5-47a2-837a-07e8bf95d745-kube-api-access-kwb5k\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.544159 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.544092 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-bundle" pod "a4357ed8-2175-40dc-b2bf-97617c6a6ad9" (UID: "a4357ed8-2175-40dc-b2bf-97617c6a6ad9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:38.544474 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.544424 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-bundle" pod "aea0bddf-6408-40e4-927a-9dfec759d091" (UID: "aea0bddf-6408-40e4-927a-9dfec759d091"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:38.545930 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.545908 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-bundle" pod "81f16422-9fd7-4e74-bb28-9410321bc55b" (UID: "81f16422-9fd7-4e74-bb28-9410321bc55b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:38.551171 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.551149 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-util" pod "a4357ed8-2175-40dc-b2bf-97617c6a6ad9" (UID: "a4357ed8-2175-40dc-b2bf-97617c6a6ad9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:38.551698 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.551671 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-util" pod "aea0bddf-6408-40e4-927a-9dfec759d091" (UID: "aea0bddf-6408-40e4-927a-9dfec759d091"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:38.551846 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.551804 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f16422-9fd7-4e74-bb28-9410321bc55b-kube-api-access-248jk" pod "81f16422-9fd7-4e74-bb28-9410321bc55b" (UID: "81f16422-9fd7-4e74-bb28-9410321bc55b"). InnerVolumeSpecName "kube-api-access-248jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:38.552086 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.552063 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea0bddf-6408-40e4-927a-9dfec759d091-kube-api-access-zw452" pod "aea0bddf-6408-40e4-927a-9dfec759d091" (UID: "aea0bddf-6408-40e4-927a-9dfec759d091"). InnerVolumeSpecName "kube-api-access-zw452". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:38.552173 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.552144 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-kube-api-access-hl8xq" pod "a4357ed8-2175-40dc-b2bf-97617c6a6ad9" (UID: "a4357ed8-2175-40dc-b2bf-97617c6a6ad9"). InnerVolumeSpecName "kube-api-access-hl8xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:38.553590 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.553563 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-util" pod "81f16422-9fd7-4e74-bb28-9410321bc55b" (UID: "81f16422-9fd7-4e74-bb28-9410321bc55b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:38.643567 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.643501 4859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-util\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.643567 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.643538 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw452\" (UniqueName: \"kubernetes.io/projected/aea0bddf-6408-40e4-927a-9dfec759d091-kube-api-access-zw452\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.643567 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.643553 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl8xq\" (UniqueName: \"kubernetes.io/projected/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-kube-api-access-hl8xq\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.643567 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.643566 4859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-util\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.643802 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.643583 4859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-util\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.643802 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.643597 4859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4357ed8-2175-40dc-b2bf-97617c6a6ad9-bundle\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.643802 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.643609 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-248jk\" (UniqueName: \"kubernetes.io/projected/81f16422-9fd7-4e74-bb28-9410321bc55b-kube-api-access-248jk\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.643802 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.643626 4859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81f16422-9fd7-4e74-bb28-9410321bc55b-bundle\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:38.643802 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:38.643639 4859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aea0bddf-6408-40e4-927a-9dfec759d091-bundle\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:39.096715 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.096612 4859 generic.go:356] "Generic (PLEG): container finished" podID="31448fe0-7459-49b7-a40f-7ee9376c57f3" containerID="5ef21e8c74b82ab822c8d974e18ec7ef358d0bb3696227e5586e909902859366" exitCode=0 Oct 30 21:46:39.096715 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.096661 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" event={"ID":"31448fe0-7459-49b7-a40f-7ee9376c57f3","Type":"ContainerDied","Data":"5ef21e8c74b82ab822c8d974e18ec7ef358d0bb3696227e5586e909902859366"} Oct 30 21:46:39.098766 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.098742 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" Oct 30 21:46:39.098893 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.098776 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klf24" event={"ID":"aea0bddf-6408-40e4-927a-9dfec759d091","Type":"ContainerDied","Data":"a02b4024fa50bcc379bcb93b354511d14a40abbcaaa6eeb49bd2ac70bfd2b634"} Oct 30 21:46:39.098893 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.098812 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a02b4024fa50bcc379bcb93b354511d14a40abbcaaa6eeb49bd2ac70bfd2b634" Oct 30 21:46:39.101541 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.101026 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" Oct 30 21:46:39.101541 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.101026 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfbj2p" event={"ID":"81f16422-9fd7-4e74-bb28-9410321bc55b","Type":"ContainerDied","Data":"f6e1d0a59e6477f60bc3d2cebc90f0668cb03317a3e2586f3d176df3e44d3411"} Oct 30 21:46:39.101541 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.101358 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6e1d0a59e6477f60bc3d2cebc90f0668cb03317a3e2586f3d176df3e44d3411" Oct 30 21:46:39.103879 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.103431 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" Oct 30 21:46:39.103879 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.103462 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a54f66" event={"ID":"0e8bad54-57f5-47a2-837a-07e8bf95d745","Type":"ContainerDied","Data":"1d6932f7f263b650559336d2e51072eae82482bd31435d854e36aecc32cc0fa3"} Oct 30 21:46:39.103879 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.103502 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d6932f7f263b650559336d2e51072eae82482bd31435d854e36aecc32cc0fa3" Oct 30 21:46:39.106202 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.106183 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" event={"ID":"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae","Type":"ContainerStarted","Data":"c6e8de59c2558f75225593a88b2b3c95bb12c7db81b27f2fa342bf6177fd4e3c"} Oct 30 21:46:39.108825 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.108803 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" event={"ID":"e7997836-978a-4993-af32-0631995a3099","Type":"ContainerStarted","Data":"7d60726e65318fcb10821b0b572a6475c9e622a1a4f0b9f2b6d495f031904906"} Oct 30 21:46:39.110882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.110858 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" event={"ID":"a4357ed8-2175-40dc-b2bf-97617c6a6ad9","Type":"ContainerDied","Data":"54598e72c7e378f6ab490e453614c24c26019a0abca00a6883fe5f9fa2c24a0b"} Oct 30 21:46:39.110994 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.110893 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vrhmh" Oct 30 21:46:39.110994 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.110908 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54598e72c7e378f6ab490e453614c24c26019a0abca00a6883fe5f9fa2c24a0b" Oct 30 21:46:39.113340 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.113321 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" event={"ID":"061abe43-b078-4bef-ab54-4a6dcbeb56fc","Type":"ContainerStarted","Data":"42a8ec104f28a7cf79dcfc6966a8f4ecd6b7a2b12d76df49de3e6fcbb3bbec71"} Oct 30 21:46:39.488527 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.488431 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" podStartSLOduration=2.945810431 podStartE2EDuration="32.488416828s" totalImagesPullingTime="29.542606397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:39.487297603 +0000 UTC m=+256.859924348" watchObservedRunningTime="2026-10-30 21:46:39.488416828 +0000 UTC m=+256.861043574" Oct 30 21:46:39.601143 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.601085 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" podStartSLOduration=4.740392355 podStartE2EDuration="40.601068727s" totalImagesPullingTime="35.860676372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:39.60034317 +0000 UTC m=+256.972969915" watchObservedRunningTime="2026-10-30 21:46:39.601068727 +0000 UTC m=+256.973695473" Oct 30 21:46:39.766569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:39.766462 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" podStartSLOduration=4.087459142 podStartE2EDuration="33.766446936s" totalImagesPullingTime="29.678987794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:39.764833004 +0000 UTC m=+257.137459749" watchObservedRunningTime="2026-10-30 21:46:39.766446936 +0000 UTC m=+257.139073681" Oct 30 21:46:39.874995 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:46:39.874934 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:46:40.119253 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.119226 4859 generic.go:356] "Generic (PLEG): container finished" podID="a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" containerID="c6e8de59c2558f75225593a88b2b3c95bb12c7db81b27f2fa342bf6177fd4e3c" exitCode=0 Oct 30 21:46:40.119648 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.119307 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" event={"ID":"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae","Type":"ContainerDied","Data":"c6e8de59c2558f75225593a88b2b3c95bb12c7db81b27f2fa342bf6177fd4e3c"} Oct 30 21:46:40.120790 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.120773 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 21:46:40.121411 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.121389 4859 generic.go:356] "Generic (PLEG): container finished" podID="e7997836-978a-4993-af32-0631995a3099" containerID="7d60726e65318fcb10821b0b572a6475c9e622a1a4f0b9f2b6d495f031904906" exitCode=1 Oct 30 21:46:40.121484 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.121466 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" event={"ID":"e7997836-978a-4993-af32-0631995a3099","Type":"ContainerDied","Data":"7d60726e65318fcb10821b0b572a6475c9e622a1a4f0b9f2b6d495f031904906"} Oct 30 21:46:40.123297 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.123277 4859 generic.go:356] "Generic (PLEG): container finished" podID="061abe43-b078-4bef-ab54-4a6dcbeb56fc" containerID="42a8ec104f28a7cf79dcfc6966a8f4ecd6b7a2b12d76df49de3e6fcbb3bbec71" exitCode=0 Oct 30 21:46:40.123383 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.123358 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" event={"ID":"061abe43-b078-4bef-ab54-4a6dcbeb56fc","Type":"ContainerDied","Data":"42a8ec104f28a7cf79dcfc6966a8f4ecd6b7a2b12d76df49de3e6fcbb3bbec71"} Oct 30 21:46:40.125189 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.125172 4859 generic.go:356] "Generic (PLEG): container finished" podID="63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" containerID="aff3deb78de2f189074ef4a19a6fd4c002b68ef95ea0c99b7cf3ccec0c3c522c" exitCode=0 Oct 30 21:46:40.125283 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.125257 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" event={"ID":"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3","Type":"ContainerDied","Data":"aff3deb78de2f189074ef4a19a6fd4c002b68ef95ea0c99b7cf3ccec0c3c522c"} Oct 30 21:46:40.240160 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.240136 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:40.261153 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.261122 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-util\" (UniqueName: \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-util\") pod \"31448fe0-7459-49b7-a40f-7ee9376c57f3\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " Oct 30 21:46:40.261294 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.261166 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-bundle\") pod \"31448fe0-7459-49b7-a40f-7ee9376c57f3\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " Oct 30 21:46:40.261294 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.261201 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31448fe0-7459-49b7-a40f-7ee9376c57f3-kube-api-access-88xzf\" (UniqueName: \"kubernetes.io/projected/31448fe0-7459-49b7-a40f-7ee9376c57f3-kube-api-access-88xzf\") pod \"31448fe0-7459-49b7-a40f-7ee9376c57f3\" (UID: \"31448fe0-7459-49b7-a40f-7ee9376c57f3\") " Oct 30 21:46:40.261627 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.261577 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-bundle" pod "31448fe0-7459-49b7-a40f-7ee9376c57f3" (UID: "31448fe0-7459-49b7-a40f-7ee9376c57f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:40.263308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.263285 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31448fe0-7459-49b7-a40f-7ee9376c57f3-kube-api-access-88xzf" pod "31448fe0-7459-49b7-a40f-7ee9376c57f3" (UID: "31448fe0-7459-49b7-a40f-7ee9376c57f3"). InnerVolumeSpecName "kube-api-access-88xzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:40.269411 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.269384 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-util" pod "31448fe0-7459-49b7-a40f-7ee9376c57f3" (UID: "31448fe0-7459-49b7-a40f-7ee9376c57f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:40.361798 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.361719 4859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-util\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:40.361798 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.361745 4859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31448fe0-7459-49b7-a40f-7ee9376c57f3-bundle\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:40.361798 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:40.361755 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88xzf\" (UniqueName: \"kubernetes.io/projected/31448fe0-7459-49b7-a40f-7ee9376c57f3-kube-api-access-88xzf\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.131422 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.131332 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" Oct 30 21:46:41.131422 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.131339 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jnstg" event={"ID":"31448fe0-7459-49b7-a40f-7ee9376c57f3","Type":"ContainerDied","Data":"5608df9d0355c457e1539646b344701831386fc2f603f44ca9b8a080f5a4d371"} Oct 30 21:46:41.131422 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.131398 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5608df9d0355c457e1539646b344701831386fc2f603f44ca9b8a080f5a4d371" Oct 30 21:46:41.264245 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.264223 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 21:46:41.266199 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.266176 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:41.310373 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.310350 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:41.346102 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.346083 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:41.349269 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.349252 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:41.371338 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371312 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-util\" (UniqueName: \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-util\") pod \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " Oct 30 21:46:41.371419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371354 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-util\" (UniqueName: \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-util\") pod \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " Oct 30 21:46:41.371419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371375 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-bundle\") pod \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " Oct 30 21:46:41.371419 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371398 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-bundle\") pod \"e7997836-978a-4993-af32-0631995a3099\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " Oct 30 21:46:41.371577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371422 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-bundle\") pod \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " Oct 30 21:46:41.371577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371451 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7997836-978a-4993-af32-0631995a3099-kube-api-access-l74rs\" (UniqueName: \"kubernetes.io/projected/e7997836-978a-4993-af32-0631995a3099-kube-api-access-l74rs\") pod \"e7997836-978a-4993-af32-0631995a3099\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " Oct 30 21:46:41.371577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371488 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-kube-api-access-cjx95\" (UniqueName: \"kubernetes.io/projected/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-kube-api-access-cjx95\") pod \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " Oct 30 21:46:41.371577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371513 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061abe43-b078-4bef-ab54-4a6dcbeb56fc-kube-api-access-czwfh\" (UniqueName: \"kubernetes.io/projected/061abe43-b078-4bef-ab54-4a6dcbeb56fc-kube-api-access-czwfh\") pod \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " Oct 30 21:46:41.371577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371545 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-util\" (UniqueName: \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-util\") pod \"e7997836-978a-4993-af32-0631995a3099\" (UID: \"e7997836-978a-4993-af32-0631995a3099\") " Oct 30 21:46:41.371577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371573 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-kube-api-access-9wm4d\" (UniqueName: \"kubernetes.io/projected/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-kube-api-access-9wm4d\") pod \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\" (UID: \"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3\") " Oct 30 21:46:41.371860 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371595 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-bundle\") pod \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\" (UID: \"061abe43-b078-4bef-ab54-4a6dcbeb56fc\") " Oct 30 21:46:41.371860 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.371634 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-util\" (UniqueName: \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-util\") pod \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\" (UID: \"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae\") " Oct 30 21:46:41.372379 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.372351 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-bundle" pod "63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" (UID: "63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:41.372491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.372463 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-bundle" pod "a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" (UID: "a1c8c1cc-9a21-410c-b6fe-f9bd86505cae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:41.372877 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.372842 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-bundle" pod "e7997836-978a-4993-af32-0631995a3099" (UID: "e7997836-978a-4993-af32-0631995a3099"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:41.373109 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.373089 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-bundle" pod "061abe43-b078-4bef-ab54-4a6dcbeb56fc" (UID: "061abe43-b078-4bef-ab54-4a6dcbeb56fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:41.374424 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.374397 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-kube-api-access-cjx95" pod "a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" (UID: "a1c8c1cc-9a21-410c-b6fe-f9bd86505cae"). InnerVolumeSpecName "kube-api-access-cjx95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:41.374747 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.374725 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7997836-978a-4993-af32-0631995a3099-kube-api-access-l74rs" pod "e7997836-978a-4993-af32-0631995a3099" (UID: "e7997836-978a-4993-af32-0631995a3099"). InnerVolumeSpecName "kube-api-access-l74rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:41.374878 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.374827 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061abe43-b078-4bef-ab54-4a6dcbeb56fc-kube-api-access-czwfh" pod "061abe43-b078-4bef-ab54-4a6dcbeb56fc" (UID: "061abe43-b078-4bef-ab54-4a6dcbeb56fc"). InnerVolumeSpecName "kube-api-access-czwfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:41.376119 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.376084 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-kube-api-access-9wm4d" pod "63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" (UID: "63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3"). InnerVolumeSpecName "kube-api-access-9wm4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:41.379436 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.379414 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-util" pod "63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" (UID: "63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:41.379515 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.379486 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-util" pod "061abe43-b078-4bef-ab54-4a6dcbeb56fc" (UID: "061abe43-b078-4bef-ab54-4a6dcbeb56fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:41.380322 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.380299 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-util" pod "e7997836-978a-4993-af32-0631995a3099" (UID: "e7997836-978a-4993-af32-0631995a3099"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:41.382084 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.382028 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-util" pod "a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" (UID: "a1c8c1cc-9a21-410c-b6fe-f9bd86505cae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:41.472460 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472428 4859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-util\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472460 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472454 4859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-util\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472460 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472465 4859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-bundle\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472645 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472474 4859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-bundle\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472645 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472482 4859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-bundle\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472645 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472491 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74rs\" (UniqueName: \"kubernetes.io/projected/e7997836-978a-4993-af32-0631995a3099-kube-api-access-l74rs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472645 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472500 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjx95\" (UniqueName: \"kubernetes.io/projected/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-kube-api-access-cjx95\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472645 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472510 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czwfh\" (UniqueName: \"kubernetes.io/projected/061abe43-b078-4bef-ab54-4a6dcbeb56fc-kube-api-access-czwfh\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472645 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472518 4859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7997836-978a-4993-af32-0631995a3099-util\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472645 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472527 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wm4d\" (UniqueName: \"kubernetes.io/projected/63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3-kube-api-access-9wm4d\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472645 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472536 4859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061abe43-b078-4bef-ab54-4a6dcbeb56fc-bundle\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.472645 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:41.472545 4859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1c8c1cc-9a21-410c-b6fe-f9bd86505cae-util\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:42.137940 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.137899 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" event={"ID":"a1c8c1cc-9a21-410c-b6fe-f9bd86505cae","Type":"ContainerDied","Data":"9e4b31756ee3967b65a22b1f73f319bdd949390991c2187935dfe2f499227355"} Oct 30 21:46:42.137940 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.137913 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbj78f" Oct 30 21:46:42.138472 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.137976 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e4b31756ee3967b65a22b1f73f319bdd949390991c2187935dfe2f499227355" Oct 30 21:46:42.139298 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.139279 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 21:46:42.140286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.140087 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" Oct 30 21:46:42.140286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.140094 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw" event={"ID":"e7997836-978a-4993-af32-0631995a3099","Type":"ContainerDied","Data":"c25e47f12aed0d76ee6dc6c936414a801d055bbe64511aa2117984854c2ac351"} Oct 30 21:46:42.140286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.140132 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c25e47f12aed0d76ee6dc6c936414a801d055bbe64511aa2117984854c2ac351" Oct 30 21:46:42.142163 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.142139 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" Oct 30 21:46:42.142262 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.142167 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64444ssb" event={"ID":"061abe43-b078-4bef-ab54-4a6dcbeb56fc","Type":"ContainerDied","Data":"ec9d15cde200382d143d573eb28e36efe0439fd71e481b50887cc718b976dae8"} Oct 30 21:46:42.142262 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.142220 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec9d15cde200382d143d573eb28e36efe0439fd71e481b50887cc718b976dae8" Oct 30 21:46:42.145788 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.145764 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" event={"ID":"63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3","Type":"ContainerDied","Data":"e3a30a86ed89852e64482c1902f4eef915d782e93d224fb719b7095b4af18882"} Oct 30 21:46:42.145890 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.145803 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a30a86ed89852e64482c1902f4eef915d782e93d224fb719b7095b4af18882" Oct 30 21:46:42.145890 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:42.145805 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714tbsz" Oct 30 21:46:48.482474 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482445 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf"] Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482713 4859 cpu_manager.go:414] "containerMap: removing container" podUID="a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482723 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482732 4859 cpu_manager.go:414] "containerMap: removing container" podUID="a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" containerName="extract" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482737 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" containerName="extract" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482745 4859 cpu_manager.go:414] "containerMap: removing container" podUID="a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" containerName="pull" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482751 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" containerName="pull" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482758 4859 cpu_manager.go:414] "containerMap: removing container" podUID="a4357ed8-2175-40dc-b2bf-97617c6a6ad9" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482763 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4357ed8-2175-40dc-b2bf-97617c6a6ad9" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482772 4859 cpu_manager.go:414] "containerMap: removing container" podUID="061abe43-b078-4bef-ab54-4a6dcbeb56fc" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482777 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061abe43-b078-4bef-ab54-4a6dcbeb56fc" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482785 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e7997836-978a-4993-af32-0631995a3099" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482790 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7997836-978a-4993-af32-0631995a3099" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482795 4859 cpu_manager.go:414] "containerMap: removing container" podUID="0e8bad54-57f5-47a2-837a-07e8bf95d745" containerName="pull" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482801 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8bad54-57f5-47a2-837a-07e8bf95d745" containerName="pull" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482806 4859 cpu_manager.go:414] "containerMap: removing container" podUID="a4357ed8-2175-40dc-b2bf-97617c6a6ad9" containerName="pull" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482810 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4357ed8-2175-40dc-b2bf-97617c6a6ad9" containerName="pull" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482817 4859 cpu_manager.go:414] "containerMap: removing container" podUID="aea0bddf-6408-40e4-927a-9dfec759d091" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482821 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea0bddf-6408-40e4-927a-9dfec759d091" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482826 4859 cpu_manager.go:414] "containerMap: removing container" podUID="81f16422-9fd7-4e74-bb28-9410321bc55b" containerName="extract" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482830 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f16422-9fd7-4e74-bb28-9410321bc55b" containerName="extract" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482837 4859 cpu_manager.go:414] "containerMap: removing container" podUID="81f16422-9fd7-4e74-bb28-9410321bc55b" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482842 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f16422-9fd7-4e74-bb28-9410321bc55b" containerName="util" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482847 4859 cpu_manager.go:414] "containerMap: removing container" podUID="81f16422-9fd7-4e74-bb28-9410321bc55b" containerName="pull" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482851 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f16422-9fd7-4e74-bb28-9410321bc55b" containerName="pull" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482858 4859 cpu_manager.go:414] "containerMap: removing container" podUID="a4357ed8-2175-40dc-b2bf-97617c6a6ad9" containerName="extract" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482862 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4357ed8-2175-40dc-b2bf-97617c6a6ad9" containerName="extract" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482870 4859 cpu_manager.go:414] "containerMap: removing container" podUID="aea0bddf-6408-40e4-927a-9dfec759d091" containerName="extract" Oct 30 21:46:48.482867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482874 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea0bddf-6408-40e4-927a-9dfec759d091" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482882 4859 cpu_manager.go:414] "containerMap: removing container" podUID="31448fe0-7459-49b7-a40f-7ee9376c57f3" containerName="util" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482887 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31448fe0-7459-49b7-a40f-7ee9376c57f3" containerName="util" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482893 4859 cpu_manager.go:414] "containerMap: removing container" podUID="0e8bad54-57f5-47a2-837a-07e8bf95d745" containerName="util" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482897 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8bad54-57f5-47a2-837a-07e8bf95d745" containerName="util" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482902 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e7997836-978a-4993-af32-0631995a3099" containerName="pull" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482906 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7997836-978a-4993-af32-0631995a3099" containerName="pull" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482917 4859 cpu_manager.go:414] "containerMap: removing container" podUID="31448fe0-7459-49b7-a40f-7ee9376c57f3" containerName="pull" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482921 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31448fe0-7459-49b7-a40f-7ee9376c57f3" containerName="pull" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482926 4859 cpu_manager.go:414] "containerMap: removing container" podUID="63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" containerName="pull" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482929 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" containerName="pull" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482936 4859 cpu_manager.go:414] "containerMap: removing container" podUID="aea0bddf-6408-40e4-927a-9dfec759d091" containerName="pull" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482940 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea0bddf-6408-40e4-927a-9dfec759d091" containerName="pull" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482958 4859 cpu_manager.go:414] "containerMap: removing container" podUID="31448fe0-7459-49b7-a40f-7ee9376c57f3" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482962 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31448fe0-7459-49b7-a40f-7ee9376c57f3" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482969 4859 cpu_manager.go:414] "containerMap: removing container" podUID="63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" containerName="util" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482973 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" containerName="util" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482978 4859 cpu_manager.go:414] "containerMap: removing container" podUID="0e8bad54-57f5-47a2-837a-07e8bf95d745" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482982 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8bad54-57f5-47a2-837a-07e8bf95d745" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482988 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e7997836-978a-4993-af32-0631995a3099" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482993 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7997836-978a-4993-af32-0631995a3099" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.482998 4859 cpu_manager.go:414] "containerMap: removing container" podUID="061abe43-b078-4bef-ab54-4a6dcbeb56fc" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483002 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061abe43-b078-4bef-ab54-4a6dcbeb56fc" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483007 4859 cpu_manager.go:414] "containerMap: removing container" podUID="63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483011 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483020 4859 cpu_manager.go:414] "containerMap: removing container" podUID="061abe43-b078-4bef-ab54-4a6dcbeb56fc" containerName="pull" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483023 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061abe43-b078-4bef-ab54-4a6dcbeb56fc" containerName="pull" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483075 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f16422-9fd7-4e74-bb28-9410321bc55b" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483084 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4357ed8-2175-40dc-b2bf-97617c6a6ad9" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483088 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea0bddf-6408-40e4-927a-9dfec759d091" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483096 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="31448fe0-7459-49b7-a40f-7ee9376c57f3" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483100 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8bad54-57f5-47a2-837a-07e8bf95d745" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483106 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7997836-978a-4993-af32-0631995a3099" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483110 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="061abe43-b078-4bef-ab54-4a6dcbeb56fc" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483116 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d1e6c9-dadc-4306-b1cc-aa442d4fa8f3" containerName="extract" Oct 30 21:46:48.483704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.483122 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c8c1cc-9a21-410c-b6fe-f9bd86505cae" containerName="extract" Oct 30 21:46:48.484973 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.484940 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:46:48.521854 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.521829 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blwpv\" (UniqueName: \"kubernetes.io/projected/35aba29e-f29d-4d40-af92-faaf06b021b2-kube-api-access-blwpv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pjgsf\" (UID: \"35aba29e-f29d-4d40-af92-faaf06b021b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:46:48.521987 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.521935 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35aba29e-f29d-4d40-af92-faaf06b021b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pjgsf\" (UID: \"35aba29e-f29d-4d40-af92-faaf06b021b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:46:48.537858 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.537841 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:46:48.537937 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.537883 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:46:48.586824 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.586801 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-296vm\"" Oct 30 21:46:48.588621 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.588603 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf"] Oct 30 21:46:48.622837 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.622813 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blwpv\" (UniqueName: \"kubernetes.io/projected/35aba29e-f29d-4d40-af92-faaf06b021b2-kube-api-access-blwpv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pjgsf\" (UID: \"35aba29e-f29d-4d40-af92-faaf06b021b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:46:48.622919 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.622880 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35aba29e-f29d-4d40-af92-faaf06b021b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pjgsf\" (UID: \"35aba29e-f29d-4d40-af92-faaf06b021b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:46:48.623228 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.623212 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35aba29e-f29d-4d40-af92-faaf06b021b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pjgsf\" (UID: \"35aba29e-f29d-4d40-af92-faaf06b021b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:46:48.680872 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.680852 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blwpv\" (UniqueName: \"kubernetes.io/projected/35aba29e-f29d-4d40-af92-faaf06b021b2-kube-api-access-blwpv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pjgsf\" (UID: \"35aba29e-f29d-4d40-af92-faaf06b021b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:46:48.795487 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:48.795434 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:46:49.277628 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:49.277607 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf"] Oct 30 21:46:49.280036 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:49.280004 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35aba29e_f29d_4d40_af92_faaf06b021b2.slice/crio-ab17882f7531268191ec8053989b9b66fc906362b3a2e71b0fe69d5d9e6122c9 WatchSource:0}: Error finding container ab17882f7531268191ec8053989b9b66fc906362b3a2e71b0fe69d5d9e6122c9: Status 404 returned error can't find the container with id ab17882f7531268191ec8053989b9b66fc906362b3a2e71b0fe69d5d9e6122c9 Oct 30 21:46:50.185499 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:50.185470 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" event={"ID":"35aba29e-f29d-4d40-af92-faaf06b021b2","Type":"ContainerStarted","Data":"ab17882f7531268191ec8053989b9b66fc906362b3a2e71b0fe69d5d9e6122c9"} Oct 30 21:46:52.145097 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.145058 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm"] Oct 30 21:46:52.151271 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.151247 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" Oct 30 21:46:52.207111 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.207082 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-564cj\"" Oct 30 21:46:52.207259 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.207214 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:52.207324 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.207281 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:52.207834 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.207807 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm"] Oct 30 21:46:52.247980 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.247932 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69924777-77c8-48fc-983b-db4edaa216ec-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm\" (UID: \"69924777-77c8-48fc-983b-db4edaa216ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" Oct 30 21:46:52.248103 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.248046 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69ftp\" (UniqueName: \"kubernetes.io/projected/69924777-77c8-48fc-983b-db4edaa216ec-kube-api-access-69ftp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm\" (UID: \"69924777-77c8-48fc-983b-db4edaa216ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" Oct 30 21:46:52.348675 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.348640 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69ftp\" (UniqueName: \"kubernetes.io/projected/69924777-77c8-48fc-983b-db4edaa216ec-kube-api-access-69ftp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm\" (UID: \"69924777-77c8-48fc-983b-db4edaa216ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" Oct 30 21:46:52.348821 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.348733 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69924777-77c8-48fc-983b-db4edaa216ec-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm\" (UID: \"69924777-77c8-48fc-983b-db4edaa216ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" Oct 30 21:46:52.349085 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.349068 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69924777-77c8-48fc-983b-db4edaa216ec-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm\" (UID: \"69924777-77c8-48fc-983b-db4edaa216ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" Oct 30 21:46:52.405775 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.405719 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69ftp\" (UniqueName: \"kubernetes.io/projected/69924777-77c8-48fc-983b-db4edaa216ec-kube-api-access-69ftp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm\" (UID: \"69924777-77c8-48fc-983b-db4edaa216ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" Oct 30 21:46:52.463665 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.463636 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" Oct 30 21:46:52.695466 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.695394 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28"] Oct 30 21:46:52.698120 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.698101 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28" Oct 30 21:46:52.752119 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.752087 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxm6f\" (UniqueName: \"kubernetes.io/projected/067cbc96-3d21-422f-b69f-ffa11e3c5706-kube-api-access-xxm6f\") pod \"dns-operator-controller-manager-8557cbdb84-tmw28\" (UID: \"067cbc96-3d21-422f-b69f-ffa11e3c5706\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28" Oct 30 21:46:52.754262 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.754244 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 21:46:52.754328 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.754274 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gqqxh\"" Oct 30 21:46:52.765675 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.765646 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28"] Oct 30 21:46:52.853089 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.853061 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxm6f\" (UniqueName: \"kubernetes.io/projected/067cbc96-3d21-422f-b69f-ffa11e3c5706-kube-api-access-xxm6f\") pod \"dns-operator-controller-manager-8557cbdb84-tmw28\" (UID: \"067cbc96-3d21-422f-b69f-ffa11e3c5706\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28" Oct 30 21:46:52.927917 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.927886 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxm6f\" (UniqueName: \"kubernetes.io/projected/067cbc96-3d21-422f-b69f-ffa11e3c5706-kube-api-access-xxm6f\") pod \"dns-operator-controller-manager-8557cbdb84-tmw28\" (UID: \"067cbc96-3d21-422f-b69f-ffa11e3c5706\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28" Oct 30 21:46:52.945355 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:52.945332 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm"] Oct 30 21:46:52.948376 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:52.948346 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69924777_77c8_48fc_983b_db4edaa216ec.slice/crio-1808debce841213ff2eee8f0289125f6eaf42c011109272b82aa1e9c950d8adf WatchSource:0}: Error finding container 1808debce841213ff2eee8f0289125f6eaf42c011109272b82aa1e9c950d8adf: Status 404 returned error can't find the container with id 1808debce841213ff2eee8f0289125f6eaf42c011109272b82aa1e9c950d8adf Oct 30 21:46:53.010412 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:53.010386 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28" Oct 30 21:46:53.203699 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:53.203621 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" event={"ID":"69924777-77c8-48fc-983b-db4edaa216ec","Type":"ContainerStarted","Data":"1808debce841213ff2eee8f0289125f6eaf42c011109272b82aa1e9c950d8adf"} Oct 30 21:46:53.390878 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:53.390855 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28"] Oct 30 21:46:53.393353 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:53.393323 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067cbc96_3d21_422f_b69f_ffa11e3c5706.slice/crio-3b8e3dae2d6527930e2c20e19e137801afb65f9ac9ddb925c23ac65512f4c886 WatchSource:0}: Error finding container 3b8e3dae2d6527930e2c20e19e137801afb65f9ac9ddb925c23ac65512f4c886: Status 404 returned error can't find the container with id 3b8e3dae2d6527930e2c20e19e137801afb65f9ac9ddb925c23ac65512f4c886 Oct 30 21:46:54.209755 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:54.209720 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" event={"ID":"35aba29e-f29d-4d40-af92-faaf06b021b2","Type":"ContainerStarted","Data":"125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac"} Oct 30 21:46:54.210242 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:54.209783 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:46:54.211077 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:54.211052 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28" event={"ID":"067cbc96-3d21-422f-b69f-ffa11e3c5706","Type":"ContainerStarted","Data":"3b8e3dae2d6527930e2c20e19e137801afb65f9ac9ddb925c23ac65512f4c886"} Oct 30 21:46:54.331765 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:54.331665 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" podStartSLOduration=1.6063339239999999 podStartE2EDuration="6.331654342s" totalImagesPullingTime="4.725320418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:54.331204766 +0000 UTC m=+271.703831510" watchObservedRunningTime="2026-10-30 21:46:54.331654342 +0000 UTC m=+271.704281088" Oct 30 21:46:56.223227 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:56.223188 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28" event={"ID":"067cbc96-3d21-422f-b69f-ffa11e3c5706","Type":"ContainerStarted","Data":"1ed95d118254dbe43f39377d40837a6491b85071feb827632d4a1a61e0ecf324"} Oct 30 21:46:56.223637 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:56.223257 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28" Oct 30 21:46:56.224895 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:56.224871 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" event={"ID":"69924777-77c8-48fc-983b-db4edaa216ec","Type":"ContainerStarted","Data":"b98ebb69d27d48c3617e3b52a0617a1693d7da1e6e3d6344601f23bc4e0d17e8"} Oct 30 21:46:56.372453 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:56.372408 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28" podStartSLOduration=1.693197088 podStartE2EDuration="4.372396048s" totalImagesPullingTime="2.67919896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:56.371912398 +0000 UTC m=+273.744539144" watchObservedRunningTime="2026-10-30 21:46:56.372396048 +0000 UTC m=+273.745022794" Oct 30 21:46:56.497115 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:56.497064 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jxvkm" podStartSLOduration=1.8621216120000001 podStartE2EDuration="4.497050907s" totalImagesPullingTime="2.634929295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:56.495610708 +0000 UTC m=+273.868237454" watchObservedRunningTime="2026-10-30 21:46:56.497050907 +0000 UTC m=+273.869677653" Oct 30 21:46:57.956882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:57.956842 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-llb2k"] Oct 30 21:46:57.959352 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:57.959330 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-llb2k" Oct 30 21:46:57.994891 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:57.994865 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwbn\" (UniqueName: \"kubernetes.io/projected/019fe412-f0e3-48d4-9878-963b1a067f08-kube-api-access-ttwbn\") pod \"authorino-operator-7b7bb4bc77-llb2k\" (UID: \"019fe412-f0e3-48d4-9878-963b1a067f08\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-llb2k" Oct 30 21:46:58.060927 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:58.060903 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4gvnm\"" Oct 30 21:46:58.063068 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:58.063045 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-llb2k"] Oct 30 21:46:58.095919 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:58.095893 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwbn\" (UniqueName: \"kubernetes.io/projected/019fe412-f0e3-48d4-9878-963b1a067f08-kube-api-access-ttwbn\") pod \"authorino-operator-7b7bb4bc77-llb2k\" (UID: \"019fe412-f0e3-48d4-9878-963b1a067f08\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-llb2k" Oct 30 21:46:58.152036 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:58.152015 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwbn\" (UniqueName: \"kubernetes.io/projected/019fe412-f0e3-48d4-9878-963b1a067f08-kube-api-access-ttwbn\") pod \"authorino-operator-7b7bb4bc77-llb2k\" (UID: \"019fe412-f0e3-48d4-9878-963b1a067f08\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-llb2k" Oct 30 21:46:58.270430 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:58.270368 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-llb2k" Oct 30 21:46:58.641596 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:58.641572 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-llb2k"] Oct 30 21:46:58.644112 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:46:58.644083 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019fe412_f0e3_48d4_9878_963b1a067f08.slice/crio-de3d073bc3f18aa9b446e641c16cf5eaad1b91bf899249eebeadfb56288419a4 WatchSource:0}: Error finding container de3d073bc3f18aa9b446e641c16cf5eaad1b91bf899249eebeadfb56288419a4: Status 404 returned error can't find the container with id de3d073bc3f18aa9b446e641c16cf5eaad1b91bf899249eebeadfb56288419a4 Oct 30 21:46:59.240677 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:46:59.240642 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-llb2k" event={"ID":"019fe412-f0e3-48d4-9878-963b1a067f08","Type":"ContainerStarted","Data":"de3d073bc3f18aa9b446e641c16cf5eaad1b91bf899249eebeadfb56288419a4"} Oct 30 21:47:01.680091 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:01.680007 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl"] Oct 30 21:47:01.682323 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:01.682304 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl" Oct 30 21:47:01.728832 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:01.728798 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx26w\" (UniqueName: \"kubernetes.io/projected/a300142a-c951-45c6-832a-ec9949b961d5-kube-api-access-lx26w\") pod \"limitador-operator-controller-manager-5778bb4449-z6cfl\" (UID: \"a300142a-c951-45c6-832a-ec9949b961d5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl" Oct 30 21:47:01.783545 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:01.783515 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4jlnk\"" Oct 30 21:47:01.785316 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:01.785296 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl"] Oct 30 21:47:01.830037 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:01.830007 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx26w\" (UniqueName: \"kubernetes.io/projected/a300142a-c951-45c6-832a-ec9949b961d5-kube-api-access-lx26w\") pod \"limitador-operator-controller-manager-5778bb4449-z6cfl\" (UID: \"a300142a-c951-45c6-832a-ec9949b961d5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl" Oct 30 21:47:01.887841 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:01.887814 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx26w\" (UniqueName: \"kubernetes.io/projected/a300142a-c951-45c6-832a-ec9949b961d5-kube-api-access-lx26w\") pod \"limitador-operator-controller-manager-5778bb4449-z6cfl\" (UID: \"a300142a-c951-45c6-832a-ec9949b961d5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl" Oct 30 21:47:01.994593 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:01.994518 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl" Oct 30 21:47:02.256959 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:02.256863 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-llb2k" event={"ID":"019fe412-f0e3-48d4-9878-963b1a067f08","Type":"ContainerStarted","Data":"19b440c8da454aa68b83da652f8e3d18ef3fce7e3cc8762ce0b860e752e10810"} Oct 30 21:47:02.257083 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:02.256957 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-llb2k" Oct 30 21:47:02.362695 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:02.362667 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl"] Oct 30 21:47:02.365149 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:47:02.365119 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda300142a_c951_45c6_832a_ec9949b961d5.slice/crio-ea025d8913dae0b6ec85e572cc1f9705bce4a6bf04de50eb6d8b5290507a941d WatchSource:0}: Error finding container ea025d8913dae0b6ec85e572cc1f9705bce4a6bf04de50eb6d8b5290507a941d: Status 404 returned error can't find the container with id ea025d8913dae0b6ec85e572cc1f9705bce4a6bf04de50eb6d8b5290507a941d Oct 30 21:47:02.410752 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:02.410702 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-llb2k" podStartSLOduration=2.6525172980000002 podStartE2EDuration="5.410689922s" totalImagesPullingTime="2.758172624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:02.369076962 +0000 UTC m=+279.741703719" watchObservedRunningTime="2026-10-30 21:47:02.410689922 +0000 UTC m=+279.783316666" Oct 30 21:47:03.263003 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:03.262969 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl" event={"ID":"a300142a-c951-45c6-832a-ec9949b961d5","Type":"ContainerStarted","Data":"ea025d8913dae0b6ec85e572cc1f9705bce4a6bf04de50eb6d8b5290507a941d"} Oct 30 21:47:05.218621 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.218590 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:47:05.274067 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.274039 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl" event={"ID":"a300142a-c951-45c6-832a-ec9949b961d5","Type":"ContainerStarted","Data":"a1f3a168a93855e2d9daf0e8510024fc686329b2da4e892e551099a9f6669bc9"} Oct 30 21:47:05.274218 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.274107 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl" Oct 30 21:47:05.525435 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.525360 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp"] Oct 30 21:47:05.528745 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.528730 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.560276 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.560251 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/04f5707e-88aa-419e-b4e4-185731c9f809-manifests\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.560384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.560291 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/04f5707e-88aa-419e-b4e4-185731c9f809-metrics-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.560384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.560319 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcmnp\" (UniqueName: \"kubernetes.io/projected/04f5707e-88aa-419e-b4e4-185731c9f809-kube-api-access-kcmnp\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.560476 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.560390 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04f5707e-88aa-419e-b4e4-185731c9f809-webhook-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.560561 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.560542 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/04f5707e-88aa-419e-b4e4-185731c9f809-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.560607 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.560593 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04f5707e-88aa-419e-b4e4-185731c9f809-tmp\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.563835 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.563797 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl" podStartSLOduration=2.252411487 podStartE2EDuration="4.56378661s" totalImagesPullingTime="2.311375123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:05.559003496 +0000 UTC m=+282.931630243" watchObservedRunningTime="2026-10-30 21:47:05.56378661 +0000 UTC m=+282.936413356" Oct 30 21:47:05.586749 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.586726 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:47:05.586838 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.586788 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7pn2f\"" Oct 30 21:47:05.587887 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.587870 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:47:05.588025 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.588011 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:47:05.592799 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.592771 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:47:05.647847 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.647826 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp"] Oct 30 21:47:05.661237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.661214 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/04f5707e-88aa-419e-b4e4-185731c9f809-manifests\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.661321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.661243 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/04f5707e-88aa-419e-b4e4-185731c9f809-metrics-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.661321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.661263 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcmnp\" (UniqueName: \"kubernetes.io/projected/04f5707e-88aa-419e-b4e4-185731c9f809-kube-api-access-kcmnp\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.661321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.661281 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04f5707e-88aa-419e-b4e4-185731c9f809-webhook-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.661481 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.661327 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/04f5707e-88aa-419e-b4e4-185731c9f809-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.661481 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.661382 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04f5707e-88aa-419e-b4e4-185731c9f809-tmp\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.661702 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.661679 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/04f5707e-88aa-419e-b4e4-185731c9f809-manifests\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.664068 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.664008 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/04f5707e-88aa-419e-b4e4-185731c9f809-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.664068 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.664026 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04f5707e-88aa-419e-b4e4-185731c9f809-tmp\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.664068 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.664010 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04f5707e-88aa-419e-b4e4-185731c9f809-webhook-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.684531 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.684511 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:47:05.693755 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.693729 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/04f5707e-88aa-419e-b4e4-185731c9f809-metrics-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.717324 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.717300 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcmnp\" (UniqueName: \"kubernetes.io/projected/04f5707e-88aa-419e-b4e4-185731c9f809-kube-api-access-kcmnp\") pod \"opendatahub-operator-controller-manager-578f754bb8-jpjkp\" (UID: \"04f5707e-88aa-419e-b4e4-185731c9f809\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:05.843781 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:05.843757 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:06.297282 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.297200 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn"] Oct 30 21:47:06.300685 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.300669 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" Oct 30 21:47:06.341347 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.341327 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp"] Oct 30 21:47:06.343588 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:47:06.343561 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f5707e_88aa_419e_b4e4_185731c9f809.slice/crio-3f661308ff8bf0359204d396ec8e2838c2caa7c80813c4afc5d5f32954a932a9 WatchSource:0}: Error finding container 3f661308ff8bf0359204d396ec8e2838c2caa7c80813c4afc5d5f32954a932a9: Status 404 returned error can't find the container with id 3f661308ff8bf0359204d396ec8e2838c2caa7c80813c4afc5d5f32954a932a9 Oct 30 21:47:06.361890 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.361868 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn"] Oct 30 21:47:06.366128 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.366099 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzsq\" (UniqueName: \"kubernetes.io/projected/512e3b58-f29f-404d-8fef-4675058fac54-kube-api-access-5tzsq\") pod \"kuadrant-operator-controller-manager-678d87766f-l56rn\" (UID: \"512e3b58-f29f-404d-8fef-4675058fac54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" Oct 30 21:47:06.366249 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.366162 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/512e3b58-f29f-404d-8fef-4675058fac54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l56rn\" (UID: \"512e3b58-f29f-404d-8fef-4675058fac54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" Oct 30 21:47:06.467140 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.467113 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/512e3b58-f29f-404d-8fef-4675058fac54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l56rn\" (UID: \"512e3b58-f29f-404d-8fef-4675058fac54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" Oct 30 21:47:06.467314 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.467230 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzsq\" (UniqueName: \"kubernetes.io/projected/512e3b58-f29f-404d-8fef-4675058fac54-kube-api-access-5tzsq\") pod \"kuadrant-operator-controller-manager-678d87766f-l56rn\" (UID: \"512e3b58-f29f-404d-8fef-4675058fac54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" Oct 30 21:47:06.467493 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.467473 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/512e3b58-f29f-404d-8fef-4675058fac54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l56rn\" (UID: \"512e3b58-f29f-404d-8fef-4675058fac54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" Oct 30 21:47:06.528302 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.528276 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzsq\" (UniqueName: \"kubernetes.io/projected/512e3b58-f29f-404d-8fef-4675058fac54-kube-api-access-5tzsq\") pod \"kuadrant-operator-controller-manager-678d87766f-l56rn\" (UID: \"512e3b58-f29f-404d-8fef-4675058fac54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" Oct 30 21:47:06.613011 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.612992 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" Oct 30 21:47:06.985628 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:06.985603 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn"] Oct 30 21:47:06.988058 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:47:06.988034 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512e3b58_f29f_404d_8fef_4675058fac54.slice/crio-4fd14086c9823949794f23c052f070906aa36491e0e443a72e942a1a1e30181e WatchSource:0}: Error finding container 4fd14086c9823949794f23c052f070906aa36491e0e443a72e942a1a1e30181e: Status 404 returned error can't find the container with id 4fd14086c9823949794f23c052f070906aa36491e0e443a72e942a1a1e30181e Oct 30 21:47:07.232136 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:07.232106 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmw28" Oct 30 21:47:07.286599 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:07.286517 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" event={"ID":"512e3b58-f29f-404d-8fef-4675058fac54","Type":"ContainerStarted","Data":"cd5602fac2dc8f2ac6b801f87ee3b2a8e4580e51e3e1059fdc16474d9cb5dda7"} Oct 30 21:47:07.286599 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:07.286576 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" event={"ID":"512e3b58-f29f-404d-8fef-4675058fac54","Type":"ContainerStarted","Data":"4fd14086c9823949794f23c052f070906aa36491e0e443a72e942a1a1e30181e"} Oct 30 21:47:07.286818 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:07.286654 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" Oct 30 21:47:07.288180 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:07.288151 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" event={"ID":"04f5707e-88aa-419e-b4e4-185731c9f809","Type":"ContainerStarted","Data":"3f661308ff8bf0359204d396ec8e2838c2caa7c80813c4afc5d5f32954a932a9"} Oct 30 21:47:07.476108 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:07.476022 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" podStartSLOduration=1.476001273 podStartE2EDuration="1.476001273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:07.474381016 +0000 UTC m=+284.847007764" watchObservedRunningTime="2026-10-30 21:47:07.476001273 +0000 UTC m=+284.848628020" Oct 30 21:47:09.875109 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:47:09.875030 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:47:10.307198 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:10.307114 4859 generic.go:356] "Generic (PLEG): container finished" podID="04f5707e-88aa-419e-b4e4-185731c9f809" containerID="15115f6e892cf3118e08625f4968d665127fa50669c3b4d984921e1cce9db7b4" exitCode=0 Oct 30 21:47:10.307198 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:10.307167 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" event={"ID":"04f5707e-88aa-419e-b4e4-185731c9f809","Type":"ContainerDied","Data":"15115f6e892cf3118e08625f4968d665127fa50669c3b4d984921e1cce9db7b4"} Oct 30 21:47:11.314286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:11.314251 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" event={"ID":"04f5707e-88aa-419e-b4e4-185731c9f809","Type":"ContainerStarted","Data":"7d2b918b0a85917566a60601d230d601f8a3d671971da8685f433dd437ada881"} Oct 30 21:47:11.314682 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:11.314340 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:11.470414 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:11.470363 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" podStartSLOduration=2.02750823 podStartE2EDuration="6.470349631s" totalImagesPullingTime="4.039185609s" totalInitContainerRuntime="403.655792ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:11.469253413 +0000 UTC m=+288.841880160" watchObservedRunningTime="2026-10-30 21:47:11.470349631 +0000 UTC m=+288.842976376" Oct 30 21:47:13.265233 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:13.265187 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-llb2k" Oct 30 21:47:16.281114 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:16.281079 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z6cfl" Oct 30 21:47:18.297067 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:18.297035 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l56rn" Oct 30 21:47:18.522127 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:18.522094 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf"] Oct 30 21:47:18.522333 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:18.522311 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" podUID="35aba29e-f29d-4d40-af92-faaf06b021b2" containerName="manager" containerID="cri-o://125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac" gracePeriod=10 Oct 30 21:47:18.769094 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:18.769073 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:47:18.873068 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:18.873043 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35aba29e-f29d-4d40-af92-faaf06b021b2-kube-api-access-blwpv\" (UniqueName: \"kubernetes.io/projected/35aba29e-f29d-4d40-af92-faaf06b021b2-kube-api-access-blwpv\") pod \"35aba29e-f29d-4d40-af92-faaf06b021b2\" (UID: \"35aba29e-f29d-4d40-af92-faaf06b021b2\") " Oct 30 21:47:18.873241 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:18.873086 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35aba29e-f29d-4d40-af92-faaf06b021b2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35aba29e-f29d-4d40-af92-faaf06b021b2-extensions-socket-volume\") pod \"35aba29e-f29d-4d40-af92-faaf06b021b2\" (UID: \"35aba29e-f29d-4d40-af92-faaf06b021b2\") " Oct 30 21:47:18.873610 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:18.873583 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35aba29e-f29d-4d40-af92-faaf06b021b2-extensions-socket-volume" pod "35aba29e-f29d-4d40-af92-faaf06b021b2" (UID: "35aba29e-f29d-4d40-af92-faaf06b021b2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:18.875233 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:18.875210 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35aba29e-f29d-4d40-af92-faaf06b021b2-kube-api-access-blwpv" pod "35aba29e-f29d-4d40-af92-faaf06b021b2" (UID: "35aba29e-f29d-4d40-af92-faaf06b021b2"). InnerVolumeSpecName "kube-api-access-blwpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:18.973684 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:18.973621 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blwpv\" (UniqueName: \"kubernetes.io/projected/35aba29e-f29d-4d40-af92-faaf06b021b2-kube-api-access-blwpv\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:18.973684 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:18.973646 4859 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35aba29e-f29d-4d40-af92-faaf06b021b2-extensions-socket-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:19.359180 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:19.359146 4859 generic.go:356] "Generic (PLEG): container finished" podID="35aba29e-f29d-4d40-af92-faaf06b021b2" containerID="125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac" exitCode=0 Oct 30 21:47:19.359546 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:19.359191 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" Oct 30 21:47:19.359546 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:19.359230 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" event={"ID":"35aba29e-f29d-4d40-af92-faaf06b021b2","Type":"ContainerDied","Data":"125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac"} Oct 30 21:47:19.359546 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:19.359284 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf" event={"ID":"35aba29e-f29d-4d40-af92-faaf06b021b2","Type":"ContainerDied","Data":"ab17882f7531268191ec8053989b9b66fc906362b3a2e71b0fe69d5d9e6122c9"} Oct 30 21:47:19.359546 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:19.359308 4859 scope.go:122] "RemoveContainer" containerID="125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac" Oct 30 21:47:19.368590 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:19.368575 4859 scope.go:122] "RemoveContainer" containerID="125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac" Oct 30 21:47:19.368849 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:47:19.368832 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac\": container with ID starting with 125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac not found: ID does not exist" containerID="125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac" Oct 30 21:47:19.368926 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:19.368860 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac"} err="failed to get container status \"125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac\": rpc error: code = NotFound desc = could not find container \"125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac\": container with ID starting with 125e57799c071882ed6e12547aece218a4ece8f2842bbb08b4b0aab3fc6682ac not found: ID does not exist" Oct 30 21:47:19.529448 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:19.529412 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf"] Oct 30 21:47:19.579455 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:19.579428 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pjgsf"] Oct 30 21:47:20.739154 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:20.739125 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35aba29e-f29d-4d40-af92-faaf06b021b2" path="/var/lib/kubelet/pods/35aba29e-f29d-4d40-af92-faaf06b021b2/volumes" Oct 30 21:47:22.321390 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:22.321358 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-jpjkp" Oct 30 21:47:22.770462 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:22.770442 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 21:47:22.770611 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:22.770591 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 21:47:22.777642 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:22.777624 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 21:47:22.777729 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:22.777696 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 21:47:36.172454 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.172380 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2"] Oct 30 21:47:36.172795 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.172711 4859 cpu_manager.go:414] "containerMap: removing container" podUID="35aba29e-f29d-4d40-af92-faaf06b021b2" containerName="manager" Oct 30 21:47:36.172795 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.172721 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35aba29e-f29d-4d40-af92-faaf06b021b2" containerName="manager" Oct 30 21:47:36.172862 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.172805 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="35aba29e-f29d-4d40-af92-faaf06b021b2" containerName="manager" Oct 30 21:47:36.174592 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.174570 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.226581 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.226548 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-q9286\"" Oct 30 21:47:36.277836 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.277806 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2"] Oct 30 21:47:36.307739 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.307709 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jpqb\" (UniqueName: \"kubernetes.io/projected/af869660-acc6-4f23-a134-a9688b49b17b-kube-api-access-7jpqb\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.307862 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.307742 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.307862 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.307767 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af869660-acc6-4f23-a134-a9688b49b17b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.307862 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.307838 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.307989 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.307891 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.307989 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.307920 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.307989 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.307959 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af869660-acc6-4f23-a134-a9688b49b17b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.308125 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.308029 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.308125 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.308059 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af869660-acc6-4f23-a134-a9688b49b17b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.409367 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.409334 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.409367 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.409370 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af869660-acc6-4f23-a134-a9688b49b17b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.409608 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.409391 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.409608 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.409413 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af869660-acc6-4f23-a134-a9688b49b17b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.409608 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.409463 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jpqb\" (UniqueName: \"kubernetes.io/projected/af869660-acc6-4f23-a134-a9688b49b17b-kube-api-access-7jpqb\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.409608 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.409485 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.409608 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.409512 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af869660-acc6-4f23-a134-a9688b49b17b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.409608 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.409544 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.409608 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.409573 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.409960 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.409876 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.410065 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.410015 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.410196 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.410182 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.410268 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.410210 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af869660-acc6-4f23-a134-a9688b49b17b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.410316 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.410289 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.412226 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.412201 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af869660-acc6-4f23-a134-a9688b49b17b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.412392 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.412374 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af869660-acc6-4f23-a134-a9688b49b17b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.467038 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.466934 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af869660-acc6-4f23-a134-a9688b49b17b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.468133 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.468112 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jpqb\" (UniqueName: \"kubernetes.io/projected/af869660-acc6-4f23-a134-a9688b49b17b-kube-api-access-7jpqb\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2\" (UID: \"af869660-acc6-4f23-a134-a9688b49b17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.485756 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.485735 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:36.906049 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.906017 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2"] Oct 30 21:47:36.946175 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:47:36.946137 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf869660_acc6_4f23_a134_a9688b49b17b.slice/crio-75d3cc0a60756aee46c0b670e89462fa642a6e1c3a758df79ea31c70ce7cc520 WatchSource:0}: Error finding container 75d3cc0a60756aee46c0b670e89462fa642a6e1c3a758df79ea31c70ce7cc520: Status 404 returned error can't find the container with id 75d3cc0a60756aee46c0b670e89462fa642a6e1c3a758df79ea31c70ce7cc520 Oct 30 21:47:36.948703 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.948655 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:36.948822 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.948749 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:36.948884 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:36.948819 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:37.447380 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:37.447342 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" event={"ID":"af869660-acc6-4f23-a134-a9688b49b17b","Type":"ContainerStarted","Data":"f29b70e0d61d55f7af6ba32c4467c6892a7136a379d188c8cffb54b87c064f5c"} Oct 30 21:47:37.447741 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:37.447410 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" event={"ID":"af869660-acc6-4f23-a134-a9688b49b17b","Type":"ContainerStarted","Data":"75d3cc0a60756aee46c0b670e89462fa642a6e1c3a758df79ea31c70ce7cc520"} Oct 30 21:47:37.486905 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:37.486878 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:37.491469 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:37.491443 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:37.589556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:37.589504 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" podStartSLOduration=1.589489762 podStartE2EDuration="1.589489762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:37.587935257 +0000 UTC m=+314.960562002" watchObservedRunningTime="2026-10-30 21:47:37.589489762 +0000 UTC m=+314.962116507" Oct 30 21:47:38.452386 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:38.452353 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:38.453476 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:38.453453 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2" Oct 30 21:47:39.875211 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:47:39.875180 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:47:41.766424 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:41.766393 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9cb85"] Oct 30 21:47:41.769085 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:41.769063 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9cb85" Oct 30 21:47:41.858123 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:41.858092 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sghbh\" (UniqueName: \"kubernetes.io/projected/5f2355cf-ad63-4d0a-8ace-a90a8e46f2df-kube-api-access-sghbh\") pod \"authorino-57f7f8659f-9cb85\" (UID: \"5f2355cf-ad63-4d0a-8ace-a90a8e46f2df\") " pod="kuadrant-system/authorino-57f7f8659f-9cb85" Oct 30 21:47:41.863584 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:41.863559 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lbtrb\"" Oct 30 21:47:41.865455 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:41.865431 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9cb85"] Oct 30 21:47:41.959239 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:41.959207 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sghbh\" (UniqueName: \"kubernetes.io/projected/5f2355cf-ad63-4d0a-8ace-a90a8e46f2df-kube-api-access-sghbh\") pod \"authorino-57f7f8659f-9cb85\" (UID: \"5f2355cf-ad63-4d0a-8ace-a90a8e46f2df\") " pod="kuadrant-system/authorino-57f7f8659f-9cb85" Oct 30 21:47:42.020459 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:42.020402 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sghbh\" (UniqueName: \"kubernetes.io/projected/5f2355cf-ad63-4d0a-8ace-a90a8e46f2df-kube-api-access-sghbh\") pod \"authorino-57f7f8659f-9cb85\" (UID: \"5f2355cf-ad63-4d0a-8ace-a90a8e46f2df\") " pod="kuadrant-system/authorino-57f7f8659f-9cb85" Oct 30 21:47:42.092604 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:42.092571 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9cb85" Oct 30 21:47:42.467872 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:42.467845 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9cb85"] Oct 30 21:47:42.470057 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:47:42.470027 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f2355cf_ad63_4d0a_8ace_a90a8e46f2df.slice/crio-bff2605a0d1a4ae4ca1cf4dc2521ca33702b5c5d48fae3d8b1bdb807309846d6 WatchSource:0}: Error finding container bff2605a0d1a4ae4ca1cf4dc2521ca33702b5c5d48fae3d8b1bdb807309846d6: Status 404 returned error can't find the container with id bff2605a0d1a4ae4ca1cf4dc2521ca33702b5c5d48fae3d8b1bdb807309846d6 Oct 30 21:47:42.471214 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:42.471194 4859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:47:43.454225 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.454193 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:47:43.457275 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.457253 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-frd55"] Oct 30 21:47:43.457391 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.457369 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" Oct 30 21:47:43.459185 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.459169 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-frd55" Oct 30 21:47:43.476632 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.476602 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9cb85" event={"ID":"5f2355cf-ad63-4d0a-8ace-a90a8e46f2df","Type":"ContainerStarted","Data":"bff2605a0d1a4ae4ca1cf4dc2521ca33702b5c5d48fae3d8b1bdb807309846d6"} Oct 30 21:47:43.558199 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.558170 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:47:43.558199 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.558171 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hqkzq\"" Oct 30 21:47:43.560424 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.560403 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-frd55"] Oct 30 21:47:43.562114 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.562095 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:47:43.572003 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.571982 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdqc\" (UniqueName: \"kubernetes.io/projected/00fcb8cd-3877-4d61-b7bb-86a0d2aca329-kube-api-access-sxdqc\") pod \"authorino-56c6c9ddd5-frd55\" (UID: \"00fcb8cd-3877-4d61-b7bb-86a0d2aca329\") " pod="kuadrant-system/authorino-56c6c9ddd5-frd55" Oct 30 21:47:43.572107 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.572026 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzs57\" (UniqueName: \"kubernetes.io/projected/0591cd90-06f6-40c3-a2bf-411dad660142-kube-api-access-vzs57\") pod \"limitador-limitador-7c9b87d574-4xv26\" (UID: \"0591cd90-06f6-40c3-a2bf-411dad660142\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" Oct 30 21:47:43.572186 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.572167 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0591cd90-06f6-40c3-a2bf-411dad660142-config-file\") pod \"limitador-limitador-7c9b87d574-4xv26\" (UID: \"0591cd90-06f6-40c3-a2bf-411dad660142\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" Oct 30 21:47:43.673148 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.673117 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0591cd90-06f6-40c3-a2bf-411dad660142-config-file\") pod \"limitador-limitador-7c9b87d574-4xv26\" (UID: \"0591cd90-06f6-40c3-a2bf-411dad660142\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" Oct 30 21:47:43.673290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.673170 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdqc\" (UniqueName: \"kubernetes.io/projected/00fcb8cd-3877-4d61-b7bb-86a0d2aca329-kube-api-access-sxdqc\") pod \"authorino-56c6c9ddd5-frd55\" (UID: \"00fcb8cd-3877-4d61-b7bb-86a0d2aca329\") " pod="kuadrant-system/authorino-56c6c9ddd5-frd55" Oct 30 21:47:43.673290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.673192 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzs57\" (UniqueName: \"kubernetes.io/projected/0591cd90-06f6-40c3-a2bf-411dad660142-kube-api-access-vzs57\") pod \"limitador-limitador-7c9b87d574-4xv26\" (UID: \"0591cd90-06f6-40c3-a2bf-411dad660142\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" Oct 30 21:47:43.673732 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.673709 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0591cd90-06f6-40c3-a2bf-411dad660142-config-file\") pod \"limitador-limitador-7c9b87d574-4xv26\" (UID: \"0591cd90-06f6-40c3-a2bf-411dad660142\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" Oct 30 21:47:43.730111 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.730053 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzs57\" (UniqueName: \"kubernetes.io/projected/0591cd90-06f6-40c3-a2bf-411dad660142-kube-api-access-vzs57\") pod \"limitador-limitador-7c9b87d574-4xv26\" (UID: \"0591cd90-06f6-40c3-a2bf-411dad660142\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" Oct 30 21:47:43.730220 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.730151 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdqc\" (UniqueName: \"kubernetes.io/projected/00fcb8cd-3877-4d61-b7bb-86a0d2aca329-kube-api-access-sxdqc\") pod \"authorino-56c6c9ddd5-frd55\" (UID: \"00fcb8cd-3877-4d61-b7bb-86a0d2aca329\") " pod="kuadrant-system/authorino-56c6c9ddd5-frd55" Oct 30 21:47:43.762889 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.762864 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:47:43.770429 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.770410 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" Oct 30 21:47:43.776213 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:43.776194 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-frd55" Oct 30 21:47:44.197506 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:44.197483 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-frd55"] Oct 30 21:47:44.199236 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:47:44.199208 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00fcb8cd_3877_4d61_b7bb_86a0d2aca329.slice/crio-bb0c44dd5c70745f256976cdb08f71e12c7f9cfe15ef0e65d2a84f1f0f05e9f2 WatchSource:0}: Error finding container bb0c44dd5c70745f256976cdb08f71e12c7f9cfe15ef0e65d2a84f1f0f05e9f2: Status 404 returned error can't find the container with id bb0c44dd5c70745f256976cdb08f71e12c7f9cfe15ef0e65d2a84f1f0f05e9f2 Oct 30 21:47:44.249349 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:44.249327 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:47:44.251482 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:47:44.251460 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0591cd90_06f6_40c3_a2bf_411dad660142.slice/crio-aef2f6cf57ae2c7732b8727d88fa681ec5641fbe8e18a36883129a8a7e5be205 WatchSource:0}: Error finding container aef2f6cf57ae2c7732b8727d88fa681ec5641fbe8e18a36883129a8a7e5be205: Status 404 returned error can't find the container with id aef2f6cf57ae2c7732b8727d88fa681ec5641fbe8e18a36883129a8a7e5be205 Oct 30 21:47:44.483059 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:44.482966 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-frd55" event={"ID":"00fcb8cd-3877-4d61-b7bb-86a0d2aca329","Type":"ContainerStarted","Data":"bb0c44dd5c70745f256976cdb08f71e12c7f9cfe15ef0e65d2a84f1f0f05e9f2"} Oct 30 21:47:44.484194 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:44.484166 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" event={"ID":"0591cd90-06f6-40c3-a2bf-411dad660142","Type":"ContainerStarted","Data":"aef2f6cf57ae2c7732b8727d88fa681ec5641fbe8e18a36883129a8a7e5be205"} Oct 30 21:47:46.495210 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:46.495172 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" event={"ID":"0591cd90-06f6-40c3-a2bf-411dad660142","Type":"ContainerStarted","Data":"bd3d78cdb1ea3577b86764e448a5d854d8cb78d97bd521a0b77b965d8f9202e8"} Oct 30 21:47:46.495585 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:46.495370 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" Oct 30 21:47:46.607957 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:46.607904 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" podStartSLOduration=1.584983103 podStartE2EDuration="3.607850375s" totalImagesPullingTime="2.022867272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:46.607464664 +0000 UTC m=+323.980091410" watchObservedRunningTime="2026-10-30 21:47:46.607850375 +0000 UTC m=+323.980477121" Oct 30 21:47:49.512446 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:49.512409 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9cb85" event={"ID":"5f2355cf-ad63-4d0a-8ace-a90a8e46f2df","Type":"ContainerStarted","Data":"8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7"} Oct 30 21:47:49.514386 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:49.514356 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-frd55" event={"ID":"00fcb8cd-3877-4d61-b7bb-86a0d2aca329","Type":"ContainerStarted","Data":"ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f"} Oct 30 21:47:49.624259 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:49.624202 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-9cb85" podStartSLOduration=2.191422401 podStartE2EDuration="8.624185249s" totalImagesPullingTime="6.432762848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:49.62368562 +0000 UTC m=+326.996312365" watchObservedRunningTime="2026-10-30 21:47:49.624185249 +0000 UTC m=+326.996811995" Oct 30 21:47:49.734544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:49.734494 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-frd55" podStartSLOduration=2.040779793 podStartE2EDuration="6.734480551s" totalImagesPullingTime="4.693700758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:49.734337013 +0000 UTC m=+327.106963759" watchObservedRunningTime="2026-10-30 21:47:49.734480551 +0000 UTC m=+327.107107297" Oct 30 21:47:49.763002 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:49.762908 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9cb85"] Oct 30 21:47:51.523958 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:51.523891 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-9cb85" podUID="5f2355cf-ad63-4d0a-8ace-a90a8e46f2df" containerName="authorino" containerID="cri-o://8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7" gracePeriod=30 Oct 30 21:47:51.766899 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:51.766873 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9cb85" Oct 30 21:47:51.843584 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:51.843560 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2355cf-ad63-4d0a-8ace-a90a8e46f2df-kube-api-access-sghbh\" (UniqueName: \"kubernetes.io/projected/5f2355cf-ad63-4d0a-8ace-a90a8e46f2df-kube-api-access-sghbh\") pod \"5f2355cf-ad63-4d0a-8ace-a90a8e46f2df\" (UID: \"5f2355cf-ad63-4d0a-8ace-a90a8e46f2df\") " Oct 30 21:47:51.845472 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:51.845447 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2355cf-ad63-4d0a-8ace-a90a8e46f2df-kube-api-access-sghbh" pod "5f2355cf-ad63-4d0a-8ace-a90a8e46f2df" (UID: "5f2355cf-ad63-4d0a-8ace-a90a8e46f2df"). InnerVolumeSpecName "kube-api-access-sghbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:51.944826 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:51.944800 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sghbh\" (UniqueName: \"kubernetes.io/projected/5f2355cf-ad63-4d0a-8ace-a90a8e46f2df-kube-api-access-sghbh\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:52.529547 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:52.529517 4859 generic.go:356] "Generic (PLEG): container finished" podID="5f2355cf-ad63-4d0a-8ace-a90a8e46f2df" containerID="8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7" exitCode=0 Oct 30 21:47:52.529934 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:52.529559 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9cb85" Oct 30 21:47:52.529934 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:52.529605 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9cb85" event={"ID":"5f2355cf-ad63-4d0a-8ace-a90a8e46f2df","Type":"ContainerDied","Data":"8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7"} Oct 30 21:47:52.529934 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:52.529646 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9cb85" event={"ID":"5f2355cf-ad63-4d0a-8ace-a90a8e46f2df","Type":"ContainerDied","Data":"bff2605a0d1a4ae4ca1cf4dc2521ca33702b5c5d48fae3d8b1bdb807309846d6"} Oct 30 21:47:52.529934 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:52.529663 4859 scope.go:122] "RemoveContainer" containerID="8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7" Oct 30 21:47:52.539041 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:52.539024 4859 scope.go:122] "RemoveContainer" containerID="8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7" Oct 30 21:47:52.539308 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:47:52.539289 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7\": container with ID starting with 8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7 not found: ID does not exist" containerID="8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7" Oct 30 21:47:52.539381 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:52.539321 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7"} err="failed to get container status \"8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7\": rpc error: code = NotFound desc = could not find container \"8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7\": container with ID starting with 8a6c7d04db05a966f34b999719eb7e058841656cc110d0b57cfe262ebb30aaa7 not found: ID does not exist" Oct 30 21:47:52.691431 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:52.691396 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9cb85"] Oct 30 21:47:52.698764 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:52.698739 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9cb85"] Oct 30 21:47:52.739445 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:52.739418 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2355cf-ad63-4d0a-8ace-a90a8e46f2df" path="/var/lib/kubelet/pods/5f2355cf-ad63-4d0a-8ace-a90a8e46f2df/volumes" Oct 30 21:47:57.501146 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:47:57.501117 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4xv26" Oct 30 21:48:09.875165 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:48:09.875121 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:48:39.875329 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:48:39.875284 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:48:49.964867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:49.964836 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr"] Oct 30 21:48:49.965352 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:49.965207 4859 cpu_manager.go:414] "containerMap: removing container" podUID="5f2355cf-ad63-4d0a-8ace-a90a8e46f2df" containerName="authorino" Oct 30 21:48:49.965352 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:49.965218 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2355cf-ad63-4d0a-8ace-a90a8e46f2df" containerName="authorino" Oct 30 21:48:49.965352 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:49.965272 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2355cf-ad63-4d0a-8ace-a90a8e46f2df" containerName="authorino" Oct 30 21:48:49.968552 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:49.968531 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.020522 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.020496 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 21:48:50.020640 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.020499 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 21:48:50.020640 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.020499 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:48:50.020640 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.020581 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 21:48:50.070016 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.069987 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:48:50.070016 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.070018 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-lmmc8\"" Oct 30 21:48:50.073427 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.073401 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr"] Oct 30 21:48:50.091356 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.091333 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.091465 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.091400 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.091529 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.091508 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.091564 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.091538 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzfsj\" (UniqueName: \"kubernetes.io/projected/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-kube-api-access-wzfsj\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.192807 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.192774 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.192980 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.192825 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.192980 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.192842 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfsj\" (UniqueName: \"kubernetes.io/projected/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-kube-api-access-wzfsj\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.192980 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.192874 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.193098 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:48:50.192982 4859 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 30 21:48:50.193098 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:48:50.193037 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-proxy-source podName:a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda nodeName:}" failed. No retries permitted until 2026-10-30 21:48:50.693021874 +0000 UTC m=+388.065648600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" (UID: "a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda") : configmap "usage-tenancy-proxy-source" not found Oct 30 21:48:50.193407 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.193389 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.195182 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.195163 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.250708 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.250654 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfsj\" (UniqueName: \"kubernetes.io/projected/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-kube-api-access-wzfsj\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.697541 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.697506 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.698326 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.698306 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qf8tr\" (UID: \"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:50.881630 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:50.881596 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:48:51.251610 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:51.251585 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr"] Oct 30 21:48:51.253921 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:48:51.253894 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dc8a84_5fe7_4d5d_b9e1_7891be6fbeda.slice/crio-ac02d41616fa279414564fde37d415238e4995e3ef60350d71dfd157f83c187b WatchSource:0}: Error finding container ac02d41616fa279414564fde37d415238e4995e3ef60350d71dfd157f83c187b: Status 404 returned error can't find the container with id ac02d41616fa279414564fde37d415238e4995e3ef60350d71dfd157f83c187b Oct 30 21:48:51.812702 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:51.812667 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" event={"ID":"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda","Type":"ContainerStarted","Data":"ac02d41616fa279414564fde37d415238e4995e3ef60350d71dfd157f83c187b"} Oct 30 21:48:53.190038 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.190006 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lwspj"] Oct 30 21:48:53.194683 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.194662 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" Oct 30 21:48:53.295617 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.295591 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 21:48:53.295817 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.295723 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f2745\"" Oct 30 21:48:53.297800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.297776 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lwspj"] Oct 30 21:48:53.321052 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.321027 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1-cert\") pod \"kserve-controller-manager-776cc66d47-lwspj\" (UID: \"dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" Oct 30 21:48:53.321151 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.321087 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz47b\" (UniqueName: \"kubernetes.io/projected/dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1-kube-api-access-tz47b\") pod \"kserve-controller-manager-776cc66d47-lwspj\" (UID: \"dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" Oct 30 21:48:53.416170 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.416140 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lv5fm"] Oct 30 21:48:53.419611 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.419593 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" Oct 30 21:48:53.421578 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.421551 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz47b\" (UniqueName: \"kubernetes.io/projected/dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1-kube-api-access-tz47b\") pod \"kserve-controller-manager-776cc66d47-lwspj\" (UID: \"dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" Oct 30 21:48:53.421681 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.421662 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1-cert\") pod \"kserve-controller-manager-776cc66d47-lwspj\" (UID: \"dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" Oct 30 21:48:53.424007 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.423989 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1-cert\") pod \"kserve-controller-manager-776cc66d47-lwspj\" (UID: \"dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" Oct 30 21:48:53.471630 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.471559 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-55dsr\"" Oct 30 21:48:53.471630 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.471559 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 21:48:53.477457 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.477429 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lv5fm"] Oct 30 21:48:53.479695 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.479674 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz47b\" (UniqueName: \"kubernetes.io/projected/dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1-kube-api-access-tz47b\") pod \"kserve-controller-manager-776cc66d47-lwspj\" (UID: \"dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" Oct 30 21:48:53.508377 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.508353 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" Oct 30 21:48:53.523242 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.523215 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36da7174-ce14-4f51-ae5e-21bd5b72fcae-tls-certs\") pod \"model-serving-api-75bd89fdfc-lv5fm\" (UID: \"36da7174-ce14-4f51-ae5e-21bd5b72fcae\") " pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" Oct 30 21:48:53.523357 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.523259 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdc98\" (UniqueName: \"kubernetes.io/projected/36da7174-ce14-4f51-ae5e-21bd5b72fcae-kube-api-access-jdc98\") pod \"model-serving-api-75bd89fdfc-lv5fm\" (UID: \"36da7174-ce14-4f51-ae5e-21bd5b72fcae\") " pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" Oct 30 21:48:53.624552 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.624518 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36da7174-ce14-4f51-ae5e-21bd5b72fcae-tls-certs\") pod \"model-serving-api-75bd89fdfc-lv5fm\" (UID: \"36da7174-ce14-4f51-ae5e-21bd5b72fcae\") " pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" Oct 30 21:48:53.624702 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.624559 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdc98\" (UniqueName: \"kubernetes.io/projected/36da7174-ce14-4f51-ae5e-21bd5b72fcae-kube-api-access-jdc98\") pod \"model-serving-api-75bd89fdfc-lv5fm\" (UID: \"36da7174-ce14-4f51-ae5e-21bd5b72fcae\") " pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" Oct 30 21:48:53.626904 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.626875 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36da7174-ce14-4f51-ae5e-21bd5b72fcae-tls-certs\") pod \"model-serving-api-75bd89fdfc-lv5fm\" (UID: \"36da7174-ce14-4f51-ae5e-21bd5b72fcae\") " pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" Oct 30 21:48:53.680738 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.680711 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdc98\" (UniqueName: \"kubernetes.io/projected/36da7174-ce14-4f51-ae5e-21bd5b72fcae-kube-api-access-jdc98\") pod \"model-serving-api-75bd89fdfc-lv5fm\" (UID: \"36da7174-ce14-4f51-ae5e-21bd5b72fcae\") " pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" Oct 30 21:48:53.739592 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.739510 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" Oct 30 21:48:53.882454 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:53.882431 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lwspj"] Oct 30 21:48:53.884776 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:48:53.884750 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8d7267_ae9c_4a80_b53b_565ca8f2f8c1.slice/crio-548c71fd69e985fdb3b0bf78a53e0533900d1887cb5d105d0fe816a50a05f28b WatchSource:0}: Error finding container 548c71fd69e985fdb3b0bf78a53e0533900d1887cb5d105d0fe816a50a05f28b: Status 404 returned error can't find the container with id 548c71fd69e985fdb3b0bf78a53e0533900d1887cb5d105d0fe816a50a05f28b Oct 30 21:48:54.109076 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:54.109048 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lv5fm"] Oct 30 21:48:54.111650 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:48:54.111613 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36da7174_ce14_4f51_ae5e_21bd5b72fcae.slice/crio-b44626213f039d67ed59d4e4915fe02ff030fa2c06bb44a3b62642d0073456a1 WatchSource:0}: Error finding container b44626213f039d67ed59d4e4915fe02ff030fa2c06bb44a3b62642d0073456a1: Status 404 returned error can't find the container with id b44626213f039d67ed59d4e4915fe02ff030fa2c06bb44a3b62642d0073456a1 Oct 30 21:48:54.837897 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:54.837859 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" event={"ID":"36da7174-ce14-4f51-ae5e-21bd5b72fcae","Type":"ContainerStarted","Data":"b44626213f039d67ed59d4e4915fe02ff030fa2c06bb44a3b62642d0073456a1"} Oct 30 21:48:54.840369 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:54.840313 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" event={"ID":"dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1","Type":"ContainerStarted","Data":"548c71fd69e985fdb3b0bf78a53e0533900d1887cb5d105d0fe816a50a05f28b"} Oct 30 21:48:57.867932 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:57.867630 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" event={"ID":"36da7174-ce14-4f51-ae5e-21bd5b72fcae","Type":"ContainerStarted","Data":"dda6b7d9d712ac9972fd490074c1d60fe5cd2dc35ad2c2f6cea707f21441a709"} Oct 30 21:48:57.867932 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:57.867738 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" Oct 30 21:48:57.984346 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:57.984286 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" podStartSLOduration=1.721901839 podStartE2EDuration="4.984266758s" totalImagesPullingTime="3.262364919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:57.98240165 +0000 UTC m=+395.355028396" watchObservedRunningTime="2026-10-30 21:48:57.984266758 +0000 UTC m=+395.356893505" Oct 30 21:48:58.876462 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:58.876425 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" event={"ID":"dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1","Type":"ContainerStarted","Data":"009ba3560c9032d79d8dcdfad32245d2d79dd3613dccbf64de4679714b755e56"} Oct 30 21:48:58.876892 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:58.876721 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" Oct 30 21:48:58.992627 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:48:58.992566 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" podStartSLOduration=1.408267798 podStartE2EDuration="5.992545052s" totalImagesPullingTime="4.584277254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:58.989625162 +0000 UTC m=+396.362251908" watchObservedRunningTime="2026-10-30 21:48:58.992545052 +0000 UTC m=+396.365171835" Oct 30 21:49:04.921792 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:04.921753 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" event={"ID":"a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda","Type":"ContainerStarted","Data":"517afbca2d31b800001ba8f72b6799c12259fca3d181d5fdcea086786f492c43"} Oct 30 21:49:04.922213 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:04.921916 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:49:05.038471 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:05.038372 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" podStartSLOduration=3.182112452 podStartE2EDuration="16.03835494s" totalImagesPullingTime="12.856242488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:05.037100323 +0000 UTC m=+402.409727070" watchObservedRunningTime="2026-10-30 21:49:05.03835494 +0000 UTC m=+402.410981686" Oct 30 21:49:08.885639 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:08.881725 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-lv5fm" Oct 30 21:49:09.875076 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:49:09.875044 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:49:15.932666 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:15.932634 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qf8tr" Oct 30 21:49:27.867670 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:27.867630 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:49:29.890600 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:29.890568 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lwspj" Oct 30 21:49:35.983735 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:35.983702 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6rpqq"] Oct 30 21:49:35.985799 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:35.985781 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" Oct 30 21:49:36.048688 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.048662 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6rpqq"] Oct 30 21:49:36.106506 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.106476 4859 generic.go:356] "Generic (PLEG): container finished" podID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerID="e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2" exitCode=0 Oct 30 21:49:36.106661 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.106554 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" event={"ID":"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103","Type":"ContainerDied","Data":"e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2"} Oct 30 21:49:36.106903 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.106888 4859 scope.go:122] "RemoveContainer" containerID="e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2" Oct 30 21:49:36.107962 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.107916 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:36.108026 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.107999 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:36.108060 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.108031 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:36.125034 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.125007 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79l7m\" (UniqueName: \"kubernetes.io/projected/cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27-kube-api-access-79l7m\") pod \"authorino-6f6944bcdd-6rpqq\" (UID: \"cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27\") " pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" Oct 30 21:49:36.225734 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.225581 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79l7m\" (UniqueName: \"kubernetes.io/projected/cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27-kube-api-access-79l7m\") pod \"authorino-6f6944bcdd-6rpqq\" (UID: \"cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27\") " pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" Oct 30 21:49:36.284729 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.284637 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79l7m\" (UniqueName: \"kubernetes.io/projected/cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27-kube-api-access-79l7m\") pod \"authorino-6f6944bcdd-6rpqq\" (UID: \"cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27\") " pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" Oct 30 21:49:36.296055 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.296033 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" Oct 30 21:49:36.387539 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.386994 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6rpqq"] Oct 30 21:49:36.412384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.412352 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65ff45b778-96hwh"] Oct 30 21:49:36.415325 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.415304 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65ff45b778-96hwh" Oct 30 21:49:36.472518 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.472484 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65ff45b778-96hwh"] Oct 30 21:49:36.527840 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.527808 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rswpp\" (UniqueName: \"kubernetes.io/projected/f7bd4508-323f-4313-a26f-1cd37268fecc-kube-api-access-rswpp\") pod \"authorino-65ff45b778-96hwh\" (UID: \"f7bd4508-323f-4313-a26f-1cd37268fecc\") " pod="kuadrant-system/authorino-65ff45b778-96hwh" Oct 30 21:49:36.629227 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.629192 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rswpp\" (UniqueName: \"kubernetes.io/projected/f7bd4508-323f-4313-a26f-1cd37268fecc-kube-api-access-rswpp\") pod \"authorino-65ff45b778-96hwh\" (UID: \"f7bd4508-323f-4313-a26f-1cd37268fecc\") " pod="kuadrant-system/authorino-65ff45b778-96hwh" Oct 30 21:49:36.688407 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.688379 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6rpqq"] Oct 30 21:49:36.689326 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.689306 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rswpp\" (UniqueName: \"kubernetes.io/projected/f7bd4508-323f-4313-a26f-1cd37268fecc-kube-api-access-rswpp\") pod \"authorino-65ff45b778-96hwh\" (UID: \"f7bd4508-323f-4313-a26f-1cd37268fecc\") " pod="kuadrant-system/authorino-65ff45b778-96hwh" Oct 30 21:49:36.690674 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:49:36.690651 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc81e7f2_0ce6_4d84_89c1_d1f2018bcd27.slice/crio-aa517918a8ffe2238a7b3b7b989d3ce43a2d7f2fb3ad5617c915430b084e1d47 WatchSource:0}: Error finding container aa517918a8ffe2238a7b3b7b989d3ce43a2d7f2fb3ad5617c915430b084e1d47: Status 404 returned error can't find the container with id aa517918a8ffe2238a7b3b7b989d3ce43a2d7f2fb3ad5617c915430b084e1d47 Oct 30 21:49:36.725842 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.725818 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65ff45b778-96hwh" Oct 30 21:49:36.808929 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.807484 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65ff45b778-96hwh"] Oct 30 21:49:36.837534 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.837509 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76c6d89db4-xk9l6"] Oct 30 21:49:36.840306 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.840289 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76c6d89db4-xk9l6" Oct 30 21:49:36.892480 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.892400 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 21:49:36.904890 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:36.904861 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76c6d89db4-xk9l6"] Oct 30 21:49:37.033358 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.033329 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4626969a-b09b-45d7-ac30-ba99eb3d63c1-tls-cert\") pod \"authorino-76c6d89db4-xk9l6\" (UID: \"4626969a-b09b-45d7-ac30-ba99eb3d63c1\") " pod="kuadrant-system/authorino-76c6d89db4-xk9l6" Oct 30 21:49:37.033743 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.033368 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74dvp\" (UniqueName: \"kubernetes.io/projected/4626969a-b09b-45d7-ac30-ba99eb3d63c1-kube-api-access-74dvp\") pod \"authorino-76c6d89db4-xk9l6\" (UID: \"4626969a-b09b-45d7-ac30-ba99eb3d63c1\") " pod="kuadrant-system/authorino-76c6d89db4-xk9l6" Oct 30 21:49:37.113974 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.113930 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" event={"ID":"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103","Type":"ContainerStarted","Data":"6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876"} Oct 30 21:49:37.115296 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.115264 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" event={"ID":"cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27","Type":"ContainerStarted","Data":"aa517918a8ffe2238a7b3b7b989d3ce43a2d7f2fb3ad5617c915430b084e1d47"} Oct 30 21:49:37.133727 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.133706 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65ff45b778-96hwh"] Oct 30 21:49:37.134017 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.133994 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4626969a-b09b-45d7-ac30-ba99eb3d63c1-tls-cert\") pod \"authorino-76c6d89db4-xk9l6\" (UID: \"4626969a-b09b-45d7-ac30-ba99eb3d63c1\") " pod="kuadrant-system/authorino-76c6d89db4-xk9l6" Oct 30 21:49:37.134068 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.134036 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74dvp\" (UniqueName: \"kubernetes.io/projected/4626969a-b09b-45d7-ac30-ba99eb3d63c1-kube-api-access-74dvp\") pod \"authorino-76c6d89db4-xk9l6\" (UID: \"4626969a-b09b-45d7-ac30-ba99eb3d63c1\") " pod="kuadrant-system/authorino-76c6d89db4-xk9l6" Oct 30 21:49:37.135598 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:49:37.135573 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7bd4508_323f_4313_a26f_1cd37268fecc.slice/crio-72b35eca1bf193d53c508ed453fd4ddeae7d64c28d88476081a75725cb1c90eb WatchSource:0}: Error finding container 72b35eca1bf193d53c508ed453fd4ddeae7d64c28d88476081a75725cb1c90eb: Status 404 returned error can't find the container with id 72b35eca1bf193d53c508ed453fd4ddeae7d64c28d88476081a75725cb1c90eb Oct 30 21:49:37.136497 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.136477 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4626969a-b09b-45d7-ac30-ba99eb3d63c1-tls-cert\") pod \"authorino-76c6d89db4-xk9l6\" (UID: \"4626969a-b09b-45d7-ac30-ba99eb3d63c1\") " pod="kuadrant-system/authorino-76c6d89db4-xk9l6" Oct 30 21:49:37.191581 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.191502 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74dvp\" (UniqueName: \"kubernetes.io/projected/4626969a-b09b-45d7-ac30-ba99eb3d63c1-kube-api-access-74dvp\") pod \"authorino-76c6d89db4-xk9l6\" (UID: \"4626969a-b09b-45d7-ac30-ba99eb3d63c1\") " pod="kuadrant-system/authorino-76c6d89db4-xk9l6" Oct 30 21:49:37.450475 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.450404 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76c6d89db4-xk9l6" Oct 30 21:49:37.704466 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.704380 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:49:37.704701 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.704679 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-xxpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:49:37.704753 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.704728 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Oct 30 21:49:37.854233 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:37.854205 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76c6d89db4-xk9l6"] Oct 30 21:49:37.855932 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:49:37.855900 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4626969a_b09b_45d7_ac30_ba99eb3d63c1.slice/crio-bfa32d036c56e4df49cf2c0aa039ce321987297f2f61c3c02bb27e8ee21bdf7b WatchSource:0}: Error finding container bfa32d036c56e4df49cf2c0aa039ce321987297f2f61c3c02bb27e8ee21bdf7b: Status 404 returned error can't find the container with id bfa32d036c56e4df49cf2c0aa039ce321987297f2f61c3c02bb27e8ee21bdf7b Oct 30 21:49:38.122474 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:38.122441 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76c6d89db4-xk9l6" event={"ID":"4626969a-b09b-45d7-ac30-ba99eb3d63c1","Type":"ContainerStarted","Data":"bfa32d036c56e4df49cf2c0aa039ce321987297f2f61c3c02bb27e8ee21bdf7b"} Oct 30 21:49:38.123698 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:38.123668 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65ff45b778-96hwh" event={"ID":"f7bd4508-323f-4313-a26f-1cd37268fecc","Type":"ContainerStarted","Data":"72b35eca1bf193d53c508ed453fd4ddeae7d64c28d88476081a75725cb1c90eb"} Oct 30 21:49:38.704668 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:38.704629 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-xxpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:49:38.704824 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:38.704688 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Oct 30 21:49:39.130271 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.130233 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" event={"ID":"cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27","Type":"ContainerStarted","Data":"a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a"} Oct 30 21:49:39.130811 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.130299 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" podUID="cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27" containerName="authorino" containerID="cri-o://a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a" gracePeriod=30 Oct 30 21:49:39.136342 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.136287 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65ff45b778-96hwh" event={"ID":"f7bd4508-323f-4313-a26f-1cd37268fecc","Type":"ContainerStarted","Data":"de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31"} Oct 30 21:49:39.136472 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.136360 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-65ff45b778-96hwh" podUID="f7bd4508-323f-4313-a26f-1cd37268fecc" containerName="authorino" containerID="cri-o://de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31" gracePeriod=30 Oct 30 21:49:39.257970 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.257897 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" podStartSLOduration=2.449325335 podStartE2EDuration="4.257882884s" totalImagesPullingTime="1.808557549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.256359679 +0000 UTC m=+436.628986426" watchObservedRunningTime="2026-10-30 21:49:39.257882884 +0000 UTC m=+436.630509629" Oct 30 21:49:39.507739 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.507713 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" Oct 30 21:49:39.510887 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.510840 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65ff45b778-96hwh" Oct 30 21:49:39.624848 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.624799 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-65ff45b778-96hwh" podStartSLOduration=2.20576409 podStartE2EDuration="3.62478222s" totalImagesPullingTime="1.41901813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.389288588 +0000 UTC m=+436.761915333" watchObservedRunningTime="2026-10-30 21:49:39.62478222 +0000 UTC m=+436.997408968" Oct 30 21:49:39.655458 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.655380 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27-kube-api-access-79l7m\" (UniqueName: \"kubernetes.io/projected/cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27-kube-api-access-79l7m\") pod \"cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27\" (UID: \"cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27\") " Oct 30 21:49:39.655618 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.655481 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bd4508-323f-4313-a26f-1cd37268fecc-kube-api-access-rswpp\" (UniqueName: \"kubernetes.io/projected/f7bd4508-323f-4313-a26f-1cd37268fecc-kube-api-access-rswpp\") pod \"f7bd4508-323f-4313-a26f-1cd37268fecc\" (UID: \"f7bd4508-323f-4313-a26f-1cd37268fecc\") " Oct 30 21:49:39.657402 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.657375 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bd4508-323f-4313-a26f-1cd37268fecc-kube-api-access-rswpp" pod "f7bd4508-323f-4313-a26f-1cd37268fecc" (UID: "f7bd4508-323f-4313-a26f-1cd37268fecc"). InnerVolumeSpecName "kube-api-access-rswpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:39.657503 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.657410 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27-kube-api-access-79l7m" pod "cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27" (UID: "cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27"). InnerVolumeSpecName "kube-api-access-79l7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:39.704686 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.704650 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-xxpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:49:39.704836 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.704700 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Oct 30 21:49:39.756981 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.756910 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79l7m\" (UniqueName: \"kubernetes.io/projected/cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27-kube-api-access-79l7m\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.756981 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:39.756984 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rswpp\" (UniqueName: \"kubernetes.io/projected/f7bd4508-323f-4313-a26f-1cd37268fecc-kube-api-access-rswpp\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.898274 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:49:39.898238 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:49:40.143960 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.143912 4859 generic.go:356] "Generic (PLEG): container finished" podID="cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27" containerID="a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a" exitCode=0 Oct 30 21:49:40.144438 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.143993 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" Oct 30 21:49:40.144438 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.144004 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" event={"ID":"cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27","Type":"ContainerDied","Data":"a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a"} Oct 30 21:49:40.144438 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.144068 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-6rpqq" event={"ID":"cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27","Type":"ContainerDied","Data":"aa517918a8ffe2238a7b3b7b989d3ce43a2d7f2fb3ad5617c915430b084e1d47"} Oct 30 21:49:40.144438 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.144123 4859 scope.go:122] "RemoveContainer" containerID="a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a" Oct 30 21:49:40.145784 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.145762 4859 generic.go:356] "Generic (PLEG): container finished" podID="f7bd4508-323f-4313-a26f-1cd37268fecc" containerID="de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31" exitCode=0 Oct 30 21:49:40.145883 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.145813 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65ff45b778-96hwh" Oct 30 21:49:40.146180 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.146149 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65ff45b778-96hwh" event={"ID":"f7bd4508-323f-4313-a26f-1cd37268fecc","Type":"ContainerDied","Data":"de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31"} Oct 30 21:49:40.146332 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.146224 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65ff45b778-96hwh" event={"ID":"f7bd4508-323f-4313-a26f-1cd37268fecc","Type":"ContainerDied","Data":"72b35eca1bf193d53c508ed453fd4ddeae7d64c28d88476081a75725cb1c90eb"} Oct 30 21:49:40.147753 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.147728 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76c6d89db4-xk9l6" event={"ID":"4626969a-b09b-45d7-ac30-ba99eb3d63c1","Type":"ContainerStarted","Data":"f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4"} Oct 30 21:49:40.153488 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.153465 4859 scope.go:122] "RemoveContainer" containerID="a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a" Oct 30 21:49:40.153757 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:49:40.153732 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a\": container with ID starting with a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a not found: ID does not exist" containerID="a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a" Oct 30 21:49:40.153827 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.153764 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a"} err="failed to get container status \"a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a\": rpc error: code = NotFound desc = could not find container \"a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a\": container with ID starting with a4b2a719b55f87282aecb38a4a250d990b89564e6a875a66dcd41f0e1a00776a not found: ID does not exist" Oct 30 21:49:40.153827 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.153788 4859 scope.go:122] "RemoveContainer" containerID="de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31" Oct 30 21:49:40.192134 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.191312 4859 scope.go:122] "RemoveContainer" containerID="de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31" Oct 30 21:49:40.192134 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:49:40.191715 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31\": container with ID starting with de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31 not found: ID does not exist" containerID="de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31" Oct 30 21:49:40.192134 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.191753 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31"} err="failed to get container status \"de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31\": rpc error: code = NotFound desc = could not find container \"de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31\": container with ID starting with de3d3e8c3b42ee7dc25b0a5565f1ae72aef9c825f88f450b641feffb92eb2d31 not found: ID does not exist" Oct 30 21:49:40.323237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.323176 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-76c6d89db4-xk9l6" podStartSLOduration=2.926405703 podStartE2EDuration="4.323157266s" totalImagesPullingTime="1.396751563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:40.319565226 +0000 UTC m=+437.692191971" watchObservedRunningTime="2026-10-30 21:49:40.323157266 +0000 UTC m=+437.695784018" Oct 30 21:49:40.347433 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.347402 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-frd55"] Oct 30 21:49:40.347609 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.347580 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-frd55" podUID="00fcb8cd-3877-4d61-b7bb-86a0d2aca329" containerName="authorino" containerID="cri-o://ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f" gracePeriod=30 Oct 30 21:49:40.458103 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.458022 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6rpqq"] Oct 30 21:49:40.490208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.489892 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6rpqq"] Oct 30 21:49:40.578965 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.578795 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65ff45b778-96hwh"] Oct 30 21:49:40.594212 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.594175 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65ff45b778-96hwh"] Oct 30 21:49:40.643827 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.643800 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-frd55" Oct 30 21:49:40.690521 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.690481 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fcb8cd-3877-4d61-b7bb-86a0d2aca329-kube-api-access-sxdqc\" (UniqueName: \"kubernetes.io/projected/00fcb8cd-3877-4d61-b7bb-86a0d2aca329-kube-api-access-sxdqc\") pod \"00fcb8cd-3877-4d61-b7bb-86a0d2aca329\" (UID: \"00fcb8cd-3877-4d61-b7bb-86a0d2aca329\") " Oct 30 21:49:40.692498 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.692463 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fcb8cd-3877-4d61-b7bb-86a0d2aca329-kube-api-access-sxdqc" pod "00fcb8cd-3877-4d61-b7bb-86a0d2aca329" (UID: "00fcb8cd-3877-4d61-b7bb-86a0d2aca329"). InnerVolumeSpecName "kube-api-access-sxdqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:40.704868 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.704837 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-xxpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:49:40.704964 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.704884 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Oct 30 21:49:40.739625 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.739564 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27" path="/var/lib/kubelet/pods/cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27/volumes" Oct 30 21:49:40.739891 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.739879 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bd4508-323f-4313-a26f-1cd37268fecc" path="/var/lib/kubelet/pods/f7bd4508-323f-4313-a26f-1cd37268fecc/volumes" Oct 30 21:49:40.791594 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:40.791562 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxdqc\" (UniqueName: \"kubernetes.io/projected/00fcb8cd-3877-4d61-b7bb-86a0d2aca329-kube-api-access-sxdqc\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:41.153643 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.153612 4859 generic.go:356] "Generic (PLEG): container finished" podID="00fcb8cd-3877-4d61-b7bb-86a0d2aca329" containerID="ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f" exitCode=0 Oct 30 21:49:41.154095 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.153658 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-frd55" Oct 30 21:49:41.154095 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.153691 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-frd55" event={"ID":"00fcb8cd-3877-4d61-b7bb-86a0d2aca329","Type":"ContainerDied","Data":"ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f"} Oct 30 21:49:41.154095 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.153727 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-frd55" event={"ID":"00fcb8cd-3877-4d61-b7bb-86a0d2aca329","Type":"ContainerDied","Data":"bb0c44dd5c70745f256976cdb08f71e12c7f9cfe15ef0e65d2a84f1f0f05e9f2"} Oct 30 21:49:41.154095 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.153744 4859 scope.go:122] "RemoveContainer" containerID="ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f" Oct 30 21:49:41.186936 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.186911 4859 scope.go:122] "RemoveContainer" containerID="ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f" Oct 30 21:49:41.187306 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:49:41.187285 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f\": container with ID starting with ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f not found: ID does not exist" containerID="ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f" Oct 30 21:49:41.187373 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.187316 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f"} err="failed to get container status \"ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f\": rpc error: code = NotFound desc = could not find container \"ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f\": container with ID starting with ac8fe64cd44125914dc94c62390e6ba314ce2c65f840a61dd17cb96b054f712f not found: ID does not exist" Oct 30 21:49:41.318892 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.318857 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-frd55"] Oct 30 21:49:41.324566 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.324541 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-frd55"] Oct 30 21:49:41.707136 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.707097 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-xxpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:49:41.707310 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:41.707157 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Oct 30 21:49:42.708090 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:42.708059 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:49:42.708542 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:42.708449 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:49:42.709113 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:42.709093 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:49:42.739320 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:49:42.739294 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fcb8cd-3877-4d61-b7bb-86a0d2aca329" path="/var/lib/kubelet/pods/00fcb8cd-3877-4d61-b7bb-86a0d2aca329/volumes" Oct 30 21:50:09.875967 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:50:09.875916 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:50:39.876009 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:50:39.875912 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:51:09.875398 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:51:09.875349 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:51:28.439963 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:28.439912 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:51:35.786696 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.786650 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8"] Oct 30 21:51:35.787290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.787110 4859 cpu_manager.go:414] "containerMap: removing container" podUID="00fcb8cd-3877-4d61-b7bb-86a0d2aca329" containerName="authorino" Oct 30 21:51:35.787290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.787125 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fcb8cd-3877-4d61-b7bb-86a0d2aca329" containerName="authorino" Oct 30 21:51:35.787290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.787140 4859 cpu_manager.go:414] "containerMap: removing container" podUID="cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27" containerName="authorino" Oct 30 21:51:35.787290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.787145 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27" containerName="authorino" Oct 30 21:51:35.787290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.787152 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f7bd4508-323f-4313-a26f-1cd37268fecc" containerName="authorino" Oct 30 21:51:35.787290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.787159 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bd4508-323f-4313-a26f-1cd37268fecc" containerName="authorino" Oct 30 21:51:35.787290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.787242 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fcb8cd-3877-4d61-b7bb-86a0d2aca329" containerName="authorino" Oct 30 21:51:35.787290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.787251 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc81e7f2-0ce6-4d84-89c1-d1f2018bcd27" containerName="authorino" Oct 30 21:51:35.787290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.787259 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bd4508-323f-4313-a26f-1cd37268fecc" containerName="authorino" Oct 30 21:51:35.790259 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.790236 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:35.841960 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.841916 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:51:35.842165 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.842019 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:51:35.890556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.890493 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dbz24\"" Oct 30 21:51:35.891496 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.891474 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 21:51:35.894602 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.894576 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8"] Oct 30 21:51:35.950423 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.950386 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29388185-34cf-4bb4-b6f3-331e792795c2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:35.950423 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.950425 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjx2q\" (UniqueName: \"kubernetes.io/projected/29388185-34cf-4bb4-b6f3-331e792795c2-kube-api-access-cjx2q\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:35.950620 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.950445 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:35.950620 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.950514 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:35.950620 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.950572 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:35.950716 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.950664 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:35.966564 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:35.966534 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:51:36.051690 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.051591 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29388185-34cf-4bb4-b6f3-331e792795c2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.051690 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.051636 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjx2q\" (UniqueName: \"kubernetes.io/projected/29388185-34cf-4bb4-b6f3-331e792795c2-kube-api-access-cjx2q\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.051690 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.051661 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.051978 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.051699 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.051978 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.051720 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.051978 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.051768 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.052642 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.052609 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.052813 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.052788 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.052935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.052917 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.054578 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.054555 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29388185-34cf-4bb4-b6f3-331e792795c2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.054701 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.054687 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29388185-34cf-4bb4-b6f3-331e792795c2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.110151 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.110112 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjx2q\" (UniqueName: \"kubernetes.io/projected/29388185-34cf-4bb4-b6f3-331e792795c2-kube-api-access-cjx2q\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kvcc8\" (UID: \"29388185-34cf-4bb4-b6f3-331e792795c2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.403824 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.403785 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:36.802715 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:36.802688 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8"] Oct 30 21:51:36.804697 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:51:36.804658 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29388185_34cf_4bb4_b6f3_331e792795c2.slice/crio-98eb90f49c7c579d903c719b6d3f46be6991f1ef9fe06c9f4c5ad03075014327 WatchSource:0}: Error finding container 98eb90f49c7c579d903c719b6d3f46be6991f1ef9fe06c9f4c5ad03075014327: Status 404 returned error can't find the container with id 98eb90f49c7c579d903c719b6d3f46be6991f1ef9fe06c9f4c5ad03075014327 Oct 30 21:51:37.769147 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:37.769062 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" event={"ID":"29388185-34cf-4bb4-b6f3-331e792795c2","Type":"ContainerStarted","Data":"98eb90f49c7c579d903c719b6d3f46be6991f1ef9fe06c9f4c5ad03075014327"} Oct 30 21:51:38.779804 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:38.779766 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" event={"ID":"29388185-34cf-4bb4-b6f3-331e792795c2","Type":"ContainerStarted","Data":"bfa8d790cc083b9fd59377fc590bf45f7b11782d0b7085d05ab374d6ba515b58"} Oct 30 21:51:38.897927 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:38.897873 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" podStartSLOduration=2.441930712 podStartE2EDuration="3.897860169s" totalImagesPullingTime="1.455929457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:38.895891342 +0000 UTC m=+556.268518088" watchObservedRunningTime="2026-10-30 21:51:38.897860169 +0000 UTC m=+556.270486914" Oct 30 21:51:38.947276 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:38.947244 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7"] Oct 30 21:51:38.951001 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:38.950978 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.003540 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.003508 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 21:51:39.013405 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.013382 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7"] Oct 30 21:51:39.080871 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.080767 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.080871 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.080823 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c252\" (UniqueName: \"kubernetes.io/projected/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-kube-api-access-5c252\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.081220 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.080927 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.081220 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.081004 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.081220 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.081050 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.081220 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.081093 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.182486 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.182452 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c252\" (UniqueName: \"kubernetes.io/projected/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-kube-api-access-5c252\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.182678 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.182493 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.182678 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.182525 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.182678 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.182561 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.182678 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.182602 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.182678 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.182676 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.182938 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.182912 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.183020 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.182957 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.183075 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.183029 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.184957 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.184918 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.185259 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.185234 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.240618 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.240585 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c252\" (UniqueName: \"kubernetes.io/projected/b6dfb179-54bc-4c3a-92ec-9bd5ab387460-kube-api-access-5c252\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-874n7\" (UID: \"b6dfb179-54bc-4c3a-92ec-9bd5ab387460\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.268333 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.268298 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:39.404513 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.404482 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:39.649922 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.649894 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7"] Oct 30 21:51:39.652252 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:51:39.652224 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6dfb179_54bc_4c3a_92ec_9bd5ab387460.slice/crio-813e7cf25a6225c97d8721b4b7abc38666d7beeed0fda73b66095e2acc659e26 WatchSource:0}: Error finding container 813e7cf25a6225c97d8721b4b7abc38666d7beeed0fda73b66095e2acc659e26: Status 404 returned error can't find the container with id 813e7cf25a6225c97d8721b4b7abc38666d7beeed0fda73b66095e2acc659e26 Oct 30 21:51:39.786711 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:39.786678 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" event={"ID":"b6dfb179-54bc-4c3a-92ec-9bd5ab387460","Type":"ContainerStarted","Data":"813e7cf25a6225c97d8721b4b7abc38666d7beeed0fda73b66095e2acc659e26"} Oct 30 21:51:39.875717 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:51:39.875678 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:51:40.793052 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:40.793011 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" event={"ID":"b6dfb179-54bc-4c3a-92ec-9bd5ab387460","Type":"ContainerStarted","Data":"574f1826337388c9c0c547430689bfeeb4fa019d62e83e84190acf648d5ace57"} Oct 30 21:51:40.911451 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:40.911400 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" podStartSLOduration=2.301377248 podStartE2EDuration="2.911385572s" totalImagesPullingTime="610.008324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:40.909381933 +0000 UTC m=+558.282008678" watchObservedRunningTime="2026-10-30 21:51:40.911385572 +0000 UTC m=+558.284012315" Oct 30 21:51:41.268801 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:41.268754 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:46.404423 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:46.404391 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:46.416769 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:46.416742 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:46.835589 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:46.835510 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kvcc8" Oct 30 21:51:48.900020 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:48.899983 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:51:49.269518 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.269425 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:49.283586 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.283553 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:49.551967 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.551855 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n"] Oct 30 21:51:49.556239 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.556217 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.583210 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.583174 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.583349 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.583220 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a896d62-f666-411c-9a73-e8bb1ac18267-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.583349 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.583259 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.583349 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.583303 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.583525 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.583352 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqh4\" (UniqueName: \"kubernetes.io/projected/0a896d62-f666-411c-9a73-e8bb1ac18267-kube-api-access-8wqh4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.583525 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.583393 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.611824 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.611795 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 21:51:49.615163 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.615135 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n"] Oct 30 21:51:49.684470 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.684430 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqh4\" (UniqueName: \"kubernetes.io/projected/0a896d62-f666-411c-9a73-e8bb1ac18267-kube-api-access-8wqh4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.684627 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.684585 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.684676 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.684625 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.684785 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.684756 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a896d62-f666-411c-9a73-e8bb1ac18267-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.684876 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.684829 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.684935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.684891 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.685114 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.685093 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.685279 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.685256 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.685533 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.685512 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.686897 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.686876 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a896d62-f666-411c-9a73-e8bb1ac18267-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.687338 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.687314 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a896d62-f666-411c-9a73-e8bb1ac18267-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.742857 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.742809 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqh4\" (UniqueName: \"kubernetes.io/projected/0a896d62-f666-411c-9a73-e8bb1ac18267-kube-api-access-8wqh4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n\" (UID: \"0a896d62-f666-411c-9a73-e8bb1ac18267\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:49.864787 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.864756 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-874n7" Oct 30 21:51:49.873000 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:49.872940 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:50.279675 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:50.279644 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n"] Oct 30 21:51:50.281206 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:51:50.281079 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a896d62_f666_411c_9a73_e8bb1ac18267.slice/crio-c9de8e4a18dd39e15834d9fdb77a1e57a3d0cfc03fa2cf7bcc13d215ef77724b WatchSource:0}: Error finding container c9de8e4a18dd39e15834d9fdb77a1e57a3d0cfc03fa2cf7bcc13d215ef77724b: Status 404 returned error can't find the container with id c9de8e4a18dd39e15834d9fdb77a1e57a3d0cfc03fa2cf7bcc13d215ef77724b Oct 30 21:51:50.854237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:50.854198 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" event={"ID":"0a896d62-f666-411c-9a73-e8bb1ac18267","Type":"ContainerStarted","Data":"c9de8e4a18dd39e15834d9fdb77a1e57a3d0cfc03fa2cf7bcc13d215ef77724b"} Oct 30 21:51:51.860787 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:51.860753 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" event={"ID":"0a896d62-f666-411c-9a73-e8bb1ac18267","Type":"ContainerStarted","Data":"62beff29485abb40dd841728d02aeb898996f23eb9f958ca687bc68b193c6726"} Oct 30 21:51:51.874066 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:51.874039 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:51.979752 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:51.979702 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" podStartSLOduration=2.382513599 podStartE2EDuration="2.979689648s" totalImagesPullingTime="597.176049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:51.978746619 +0000 UTC m=+569.351373363" watchObservedRunningTime="2026-10-30 21:51:51.979689648 +0000 UTC m=+569.352316443" Oct 30 21:51:52.967543 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:52.967509 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq"] Oct 30 21:51:52.971520 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:52.971500 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.015286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.015257 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.015427 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.015317 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27da8df3-0147-4496-9997-2e55b71fa988-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.015474 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.015435 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.015519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.015487 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.015617 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.015593 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.015694 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.015639 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbc28\" (UniqueName: \"kubernetes.io/projected/27da8df3-0147-4496-9997-2e55b71fa988-kube-api-access-vbc28\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.026625 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.026604 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 21:51:53.078135 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.078106 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq"] Oct 30 21:51:53.116369 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.116338 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.116565 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.116381 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.116797 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.116768 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.116870 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.116844 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.116925 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.116868 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbc28\" (UniqueName: \"kubernetes.io/projected/27da8df3-0147-4496-9997-2e55b71fa988-kube-api-access-vbc28\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.116925 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.116902 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.117052 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.116970 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27da8df3-0147-4496-9997-2e55b71fa988-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.117052 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.116992 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.117409 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.117391 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.119179 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.119150 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27da8df3-0147-4496-9997-2e55b71fa988-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.119829 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.119804 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27da8df3-0147-4496-9997-2e55b71fa988-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.175900 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.175876 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbc28\" (UniqueName: \"kubernetes.io/projected/27da8df3-0147-4496-9997-2e55b71fa988-kube-api-access-vbc28\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq\" (UID: \"27da8df3-0147-4496-9997-2e55b71fa988\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.284530 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.284441 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:53.664230 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:51:53.664203 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27da8df3_0147_4496_9997_2e55b71fa988.slice/crio-e56271f663b340f1d6a01a4ff76727f1dd9d4d757f46dde4039cb3ecbf0f9ca8 WatchSource:0}: Error finding container e56271f663b340f1d6a01a4ff76727f1dd9d4d757f46dde4039cb3ecbf0f9ca8: Status 404 returned error can't find the container with id e56271f663b340f1d6a01a4ff76727f1dd9d4d757f46dde4039cb3ecbf0f9ca8 Oct 30 21:51:53.664343 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.664304 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq"] Oct 30 21:51:53.875706 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:53.875675 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" event={"ID":"27da8df3-0147-4496-9997-2e55b71fa988","Type":"ContainerStarted","Data":"e56271f663b340f1d6a01a4ff76727f1dd9d4d757f46dde4039cb3ecbf0f9ca8"} Oct 30 21:51:54.883102 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:54.883062 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" event={"ID":"27da8df3-0147-4496-9997-2e55b71fa988","Type":"ContainerStarted","Data":"6fffb58a654ae97656c610fc704b1908a89c39b7f44f79e1ba901e861b1ec8e9"} Oct 30 21:51:54.999841 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:54.999788 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podStartSLOduration=2.410688531 podStartE2EDuration="2.999772406s" totalImagesPullingTime="589.083875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:54.999322389 +0000 UTC m=+572.371949136" watchObservedRunningTime="2026-10-30 21:51:54.999772406 +0000 UTC m=+572.372399151" Oct 30 21:51:55.285108 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:55.285030 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:51:59.873886 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:59.873854 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:59.885903 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:59.885877 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:51:59.924982 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:51:59.924931 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n" Oct 30 21:52:02.390287 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:02.390252 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:52:03.284737 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:03.284702 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:52:03.296867 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:03.296837 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:52:03.944905 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:03.944875 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 21:52:09.448557 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.448466 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58"] Oct 30 21:52:09.452737 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.452714 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.561703 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.561670 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rm8s\" (UniqueName: \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-kube-api-access-7rm8s\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.561843 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.561712 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.561843 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.561806 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.561926 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.561848 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.561926 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.561883 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.561926 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.561917 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b249f651-382f-4948-9fb1-455e46949f49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.562101 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.561986 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b249f651-382f-4948-9fb1-455e46949f49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.562101 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.562057 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.562101 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.562090 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.653302 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.653266 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58"] Oct 30 21:52:09.662738 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.662711 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.662891 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.662744 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b249f651-382f-4948-9fb1-455e46949f49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.662891 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.662767 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b249f651-382f-4948-9fb1-455e46949f49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.662891 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.662791 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.662891 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.662817 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.663148 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.662969 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rm8s\" (UniqueName: \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-kube-api-access-7rm8s\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.663148 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.663011 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.663148 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.663107 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.663308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.663154 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.663308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.663193 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.663308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.663213 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.663465 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.663341 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.663518 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.663502 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.664014 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.663989 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b249f651-382f-4948-9fb1-455e46949f49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.665345 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.665326 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.665475 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.665458 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b249f651-382f-4948-9fb1-455e46949f49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.718965 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.718896 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.719077 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.719003 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rm8s\" (UniqueName: \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-kube-api-access-7rm8s\") pod \"maas-default-gateway-openshift-default-694db55c87-dnh58\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.765182 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:09.765159 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:09.877923 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:52:09.877890 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:52:10.185534 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:10.185505 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58"] Oct 30 21:52:10.226803 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:52:10.226771 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb249f651_382f_4948_9fb1_455e46949f49.slice/crio-6ae2e1c03b9d54903fb357cb639c5a54817f22ed4dc8b2d11127a32dd19245e3 WatchSource:0}: Error finding container 6ae2e1c03b9d54903fb357cb639c5a54817f22ed4dc8b2d11127a32dd19245e3: Status 404 returned error can't find the container with id 6ae2e1c03b9d54903fb357cb639c5a54817f22ed4dc8b2d11127a32dd19245e3 Oct 30 21:52:10.229080 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:10.229050 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:52:10.229161 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:10.229113 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:52:10.229161 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:10.229144 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:52:10.991880 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:10.991840 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" event={"ID":"b249f651-382f-4948-9fb1-455e46949f49","Type":"ContainerStarted","Data":"02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7"} Oct 30 21:52:10.992350 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:10.991898 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" event={"ID":"b249f651-382f-4948-9fb1-455e46949f49","Type":"ContainerStarted","Data":"6ae2e1c03b9d54903fb357cb639c5a54817f22ed4dc8b2d11127a32dd19245e3"} Oct 30 21:52:11.204513 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:11.204453 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" podStartSLOduration=2.204439113 podStartE2EDuration="2.204439113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:11.15652024 +0000 UTC m=+588.529146988" watchObservedRunningTime="2026-10-30 21:52:11.204439113 +0000 UTC m=+588.577065859" Oct 30 21:52:11.784702 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:11.784666 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:11.786137 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:11.786103 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dnh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:52:11.786278 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:11.786151 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" podUID="b249f651-382f-4948-9fb1-455e46949f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:52:12.781973 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:12.781256 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dnh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:52:12.781973 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:12.781311 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" podUID="b249f651-382f-4948-9fb1-455e46949f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:52:13.766286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:13.766248 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dnh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:52:13.766561 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:13.766535 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" podUID="b249f651-382f-4948-9fb1-455e46949f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:52:14.778756 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:14.778311 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dnh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:52:14.778756 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:14.778370 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" podUID="b249f651-382f-4948-9fb1-455e46949f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:52:15.779968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:15.777313 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dnh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:52:15.779968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:15.777377 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" podUID="b249f651-382f-4948-9fb1-455e46949f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:52:16.765550 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:16.765519 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dnh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:52:16.765729 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:16.765576 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" podUID="b249f651-382f-4948-9fb1-455e46949f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:52:18.556855 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:18.556807 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:18.557372 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:18.557313 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:18.558232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:18.558211 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:52:21.521059 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:21.521023 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77dd9fff5-h8rlb"] Oct 30 21:52:21.526247 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:21.526195 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77dd9fff5-h8rlb" Oct 30 21:52:21.582021 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:21.581991 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77dd9fff5-h8rlb"] Oct 30 21:52:21.587340 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:21.587310 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl6r7\" (UniqueName: \"kubernetes.io/projected/2d74b772-c978-43f7-a84b-f23f68145de2-kube-api-access-cl6r7\") pod \"authorino-77dd9fff5-h8rlb\" (UID: \"2d74b772-c978-43f7-a84b-f23f68145de2\") " pod="kuadrant-system/authorino-77dd9fff5-h8rlb" Oct 30 21:52:21.587466 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:21.587432 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d74b772-c978-43f7-a84b-f23f68145de2-tls-cert\") pod \"authorino-77dd9fff5-h8rlb\" (UID: \"2d74b772-c978-43f7-a84b-f23f68145de2\") " pod="kuadrant-system/authorino-77dd9fff5-h8rlb" Oct 30 21:52:21.689201 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:21.689158 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6r7\" (UniqueName: \"kubernetes.io/projected/2d74b772-c978-43f7-a84b-f23f68145de2-kube-api-access-cl6r7\") pod \"authorino-77dd9fff5-h8rlb\" (UID: \"2d74b772-c978-43f7-a84b-f23f68145de2\") " pod="kuadrant-system/authorino-77dd9fff5-h8rlb" Oct 30 21:52:21.689441 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:21.689245 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d74b772-c978-43f7-a84b-f23f68145de2-tls-cert\") pod \"authorino-77dd9fff5-h8rlb\" (UID: \"2d74b772-c978-43f7-a84b-f23f68145de2\") " pod="kuadrant-system/authorino-77dd9fff5-h8rlb" Oct 30 21:52:21.692032 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:21.692004 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d74b772-c978-43f7-a84b-f23f68145de2-tls-cert\") pod \"authorino-77dd9fff5-h8rlb\" (UID: \"2d74b772-c978-43f7-a84b-f23f68145de2\") " pod="kuadrant-system/authorino-77dd9fff5-h8rlb" Oct 30 21:52:21.746890 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:21.746855 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6r7\" (UniqueName: \"kubernetes.io/projected/2d74b772-c978-43f7-a84b-f23f68145de2-kube-api-access-cl6r7\") pod \"authorino-77dd9fff5-h8rlb\" (UID: \"2d74b772-c978-43f7-a84b-f23f68145de2\") " pod="kuadrant-system/authorino-77dd9fff5-h8rlb" Oct 30 21:52:21.838320 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:21.838295 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77dd9fff5-h8rlb" Oct 30 21:52:22.210794 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:22.210765 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77dd9fff5-h8rlb"] Oct 30 21:52:22.212754 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:52:22.212723 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d74b772_c978_43f7_a84b_f23f68145de2.slice/crio-bce64b52ddc22de317587afe7cc5e773fbc2a73e348c1d992faf89cf81b272c2 WatchSource:0}: Error finding container bce64b52ddc22de317587afe7cc5e773fbc2a73e348c1d992faf89cf81b272c2: Status 404 returned error can't find the container with id bce64b52ddc22de317587afe7cc5e773fbc2a73e348c1d992faf89cf81b272c2 Oct 30 21:52:22.812440 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:22.812398 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 21:52:22.813666 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:22.813643 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 21:52:22.828367 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:22.828339 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 21:52:22.829481 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:22.829459 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 21:52:23.061523 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:23.061488 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77dd9fff5-h8rlb" event={"ID":"2d74b772-c978-43f7-a84b-f23f68145de2","Type":"ContainerStarted","Data":"bce64b52ddc22de317587afe7cc5e773fbc2a73e348c1d992faf89cf81b272c2"} Oct 30 21:52:24.068339 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.068302 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77dd9fff5-h8rlb" event={"ID":"2d74b772-c978-43f7-a84b-f23f68145de2","Type":"ContainerStarted","Data":"92679165dc1dd79be44b63b65380a58647fc34940d3dec2b9c7a69471e7238be"} Oct 30 21:52:24.224907 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.224842 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-77dd9fff5-h8rlb" podStartSLOduration=1.87882605 podStartE2EDuration="3.224823965s" totalImagesPullingTime="1.345997915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:24.224452247 +0000 UTC m=+601.597078995" watchObservedRunningTime="2026-10-30 21:52:24.224823965 +0000 UTC m=+601.597450715" Oct 30 21:52:24.281135 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.281101 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76c6d89db4-xk9l6"] Oct 30 21:52:24.281313 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.281274 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-76c6d89db4-xk9l6" podUID="4626969a-b09b-45d7-ac30-ba99eb3d63c1" containerName="authorino" containerID="cri-o://f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4" gracePeriod=30 Oct 30 21:52:24.553401 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.553331 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76c6d89db4-xk9l6" Oct 30 21:52:24.584435 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.584400 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n"] Oct 30 21:52:24.585122 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.585103 4859 cpu_manager.go:414] "containerMap: removing container" podUID="4626969a-b09b-45d7-ac30-ba99eb3d63c1" containerName="authorino" Oct 30 21:52:24.585188 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.585127 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4626969a-b09b-45d7-ac30-ba99eb3d63c1" containerName="authorino" Oct 30 21:52:24.585282 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.585270 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4626969a-b09b-45d7-ac30-ba99eb3d63c1" containerName="authorino" Oct 30 21:52:24.588906 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.588886 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.693048 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.687508 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n"] Oct 30 21:52:24.721015 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.720973 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4626969a-b09b-45d7-ac30-ba99eb3d63c1-tls-cert\" (UniqueName: \"kubernetes.io/secret/4626969a-b09b-45d7-ac30-ba99eb3d63c1-tls-cert\") pod \"4626969a-b09b-45d7-ac30-ba99eb3d63c1\" (UID: \"4626969a-b09b-45d7-ac30-ba99eb3d63c1\") " Oct 30 21:52:24.721216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.721042 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4626969a-b09b-45d7-ac30-ba99eb3d63c1-kube-api-access-74dvp\" (UniqueName: \"kubernetes.io/projected/4626969a-b09b-45d7-ac30-ba99eb3d63c1-kube-api-access-74dvp\") pod \"4626969a-b09b-45d7-ac30-ba99eb3d63c1\" (UID: \"4626969a-b09b-45d7-ac30-ba99eb3d63c1\") " Oct 30 21:52:24.721315 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.721280 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.721459 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.721382 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.721528 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.721483 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.721583 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.721522 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.721695 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.721594 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdkhn\" (UniqueName: \"kubernetes.io/projected/b6e11781-e771-4f4d-8458-3966ee881c40-kube-api-access-tdkhn\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.721695 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.721631 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e11781-e771-4f4d-8458-3966ee881c40-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.721695 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.721666 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.721848 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.721705 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e11781-e771-4f4d-8458-3966ee881c40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.721848 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.721831 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e11781-e771-4f4d-8458-3966ee881c40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.723667 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.723634 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4626969a-b09b-45d7-ac30-ba99eb3d63c1-kube-api-access-74dvp" pod "4626969a-b09b-45d7-ac30-ba99eb3d63c1" (UID: "4626969a-b09b-45d7-ac30-ba99eb3d63c1"). InnerVolumeSpecName "kube-api-access-74dvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:24.733276 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.732930 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4626969a-b09b-45d7-ac30-ba99eb3d63c1-tls-cert" pod "4626969a-b09b-45d7-ac30-ba99eb3d63c1" (UID: "4626969a-b09b-45d7-ac30-ba99eb3d63c1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:24.822930 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.822776 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkhn\" (UniqueName: \"kubernetes.io/projected/b6e11781-e771-4f4d-8458-3966ee881c40-kube-api-access-tdkhn\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.822930 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.822836 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e11781-e771-4f4d-8458-3966ee881c40-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.823201 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.823025 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.823201 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.823162 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e11781-e771-4f4d-8458-3966ee881c40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.823314 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.823255 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e11781-e771-4f4d-8458-3966ee881c40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.823436 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.823415 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.823566 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.823493 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.823625 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.823565 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.823625 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.823598 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.823797 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.823762 4859 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4626969a-b09b-45d7-ac30-ba99eb3d63c1-tls-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:24.823857 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.823811 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74dvp\" (UniqueName: \"kubernetes.io/projected/4626969a-b09b-45d7-ac30-ba99eb3d63c1-kube-api-access-74dvp\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:24.823913 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.823892 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.824088 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.824064 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.824260 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.824214 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.824430 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.824412 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.824735 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.824712 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e11781-e771-4f4d-8458-3966ee881c40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.826886 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.826864 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e11781-e771-4f4d-8458-3966ee881c40-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.827121 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.827092 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e11781-e771-4f4d-8458-3966ee881c40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.896368 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.896325 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e11781-e771-4f4d-8458-3966ee881c40-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.896575 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.896550 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdkhn\" (UniqueName: \"kubernetes.io/projected/b6e11781-e771-4f4d-8458-3966ee881c40-kube-api-access-tdkhn\") pod \"maas-default-gateway-openshift-default-694db55c87-mgw8n\" (UID: \"b6e11781-e771-4f4d-8458-3966ee881c40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:24.906130 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:24.906101 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:25.076344 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.076304 4859 generic.go:356] "Generic (PLEG): container finished" podID="4626969a-b09b-45d7-ac30-ba99eb3d63c1" containerID="f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4" exitCode=0 Oct 30 21:52:25.076822 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.076368 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76c6d89db4-xk9l6" Oct 30 21:52:25.076822 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.076389 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76c6d89db4-xk9l6" event={"ID":"4626969a-b09b-45d7-ac30-ba99eb3d63c1","Type":"ContainerDied","Data":"f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4"} Oct 30 21:52:25.076822 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.076439 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76c6d89db4-xk9l6" event={"ID":"4626969a-b09b-45d7-ac30-ba99eb3d63c1","Type":"ContainerDied","Data":"bfa32d036c56e4df49cf2c0aa039ce321987297f2f61c3c02bb27e8ee21bdf7b"} Oct 30 21:52:25.076822 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.076459 4859 scope.go:122] "RemoveContainer" containerID="f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4" Oct 30 21:52:25.087718 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.087691 4859 scope.go:122] "RemoveContainer" containerID="f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4" Oct 30 21:52:25.088288 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:52:25.088265 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4\": container with ID starting with f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4 not found: ID does not exist" containerID="f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4" Oct 30 21:52:25.088382 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.088298 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4"} err="failed to get container status \"f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4\": rpc error: code = NotFound desc = could not find container \"f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4\": container with ID starting with f4619145a65a57bdf8b719104aa4d2a380972b6d24c6bbacd4f395ba1138b8e4 not found: ID does not exist" Oct 30 21:52:25.242791 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.242709 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76c6d89db4-xk9l6"] Oct 30 21:52:25.246908 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.246877 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76c6d89db4-xk9l6"] Oct 30 21:52:25.357248 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.357211 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n"] Oct 30 21:52:25.357592 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:52:25.357567 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e11781_e771_4f4d_8458_3966ee881c40.slice/crio-f2ce0608c1223fa3e50cc2d99309769091bfc3b8c4c22079f639267a6078bd51 WatchSource:0}: Error finding container f2ce0608c1223fa3e50cc2d99309769091bfc3b8c4c22079f639267a6078bd51: Status 404 returned error can't find the container with id f2ce0608c1223fa3e50cc2d99309769091bfc3b8c4c22079f639267a6078bd51 Oct 30 21:52:25.359898 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.359862 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:52:25.359991 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.359970 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:52:25.360039 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:25.360023 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:52:26.119710 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:26.119659 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" event={"ID":"b6e11781-e771-4f4d-8458-3966ee881c40","Type":"ContainerStarted","Data":"29f6e86b400d8babcd55a79ed8d0e2c670c45c5c0ce910f5751880d3c6fafec0"} Oct 30 21:52:26.120338 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:26.120311 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" event={"ID":"b6e11781-e771-4f4d-8458-3966ee881c40","Type":"ContainerStarted","Data":"f2ce0608c1223fa3e50cc2d99309769091bfc3b8c4c22079f639267a6078bd51"} Oct 30 21:52:26.252521 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:26.252417 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" podStartSLOduration=2.252398087 podStartE2EDuration="2.252398087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:26.251080159 +0000 UTC m=+603.623706908" watchObservedRunningTime="2026-10-30 21:52:26.252398087 +0000 UTC m=+603.625024902" Oct 30 21:52:26.740862 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:26.740827 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4626969a-b09b-45d7-ac30-ba99eb3d63c1" path="/var/lib/kubelet/pods/4626969a-b09b-45d7-ac30-ba99eb3d63c1/volumes" Oct 30 21:52:26.906573 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:26.906535 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:26.908575 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:26.908548 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mgw8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:52:26.908711 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:26.908603 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" podUID="b6e11781-e771-4f4d-8458-3966ee881c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:52:27.906611 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:27.906571 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mgw8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:52:27.907049 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:27.906630 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" podUID="b6e11781-e771-4f4d-8458-3966ee881c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:52:28.906846 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:28.906799 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mgw8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:52:28.907376 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:28.906861 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" podUID="b6e11781-e771-4f4d-8458-3966ee881c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:52:29.906751 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:29.906704 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mgw8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:52:29.906981 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:29.906778 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" podUID="b6e11781-e771-4f4d-8458-3966ee881c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:52:30.907416 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:30.907371 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mgw8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:52:30.907925 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:30.907434 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" podUID="b6e11781-e771-4f4d-8458-3966ee881c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:52:31.906446 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:31.906411 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mgw8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:52:31.906622 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:31.906463 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" podUID="b6e11781-e771-4f4d-8458-3966ee881c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:52:32.906736 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:32.906699 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mgw8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:52:32.907174 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:32.906768 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" podUID="b6e11781-e771-4f4d-8458-3966ee881c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:52:33.911619 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:33.911588 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:33.912066 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:33.911971 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:33.912853 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:52:33.912834 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" Oct 30 21:52:39.876372 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:52:39.876340 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:53:09.876004 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:53:09.875970 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:53:23.286721 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.286688 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh"] Oct 30 21:53:23.291601 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.291583 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.359837 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.359811 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f20e6e5e-openshift-default-dockercfg-9gljv\"" Oct 30 21:53:23.413647 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.413616 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh"] Oct 30 21:53:23.481450 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.481415 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022d4791-881d-4f32-9531-f0ded16008f3-istiod-ca-cert\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.481612 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.481479 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-certs\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.481612 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.481544 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hvtv\" (UniqueName: \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-kube-api-access-8hvtv\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.481612 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.481582 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-socket\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.481612 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.481605 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-istio-token\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.481741 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.481706 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-envoy\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.481741 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.481731 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022d4791-881d-4f32-9531-f0ded16008f3-istio-podinfo\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.481804 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.481772 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-credential-socket\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.481852 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.481839 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-data\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583006 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.582895 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-data\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583006 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.582968 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022d4791-881d-4f32-9531-f0ded16008f3-istiod-ca-cert\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583234 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583035 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-certs\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583234 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583060 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hvtv\" (UniqueName: \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-kube-api-access-8hvtv\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583234 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583093 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-socket\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583234 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583121 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-istio-token\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583234 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583171 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-envoy\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583234 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583196 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022d4791-881d-4f32-9531-f0ded16008f3-istio-podinfo\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583264 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-credential-socket\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583341 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-data\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583451 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-certs\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583722 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022d4791-881d-4f32-9531-f0ded16008f3-istiod-ca-cert\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.583882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583819 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-socket\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.584118 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.583936 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-credential-socket\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.585739 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.585715 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-envoy\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.585938 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.585919 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022d4791-881d-4f32-9531-f0ded16008f3-istio-podinfo\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.652777 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.652753 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-istio-token\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.652898 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.652870 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hvtv\" (UniqueName: \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-kube-api-access-8hvtv\") pod \"e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:23.904675 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:23.904643 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:24.366541 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:24.366518 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh"] Oct 30 21:53:24.367805 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:53:24.367774 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022d4791_881d_4f32_9531_f0ded16008f3.slice/crio-fd2c9be6d8f74e9e87d18d4c8b55d482ceea035b00201a74c71f0254d8ee6155 WatchSource:0}: Error finding container fd2c9be6d8f74e9e87d18d4c8b55d482ceea035b00201a74c71f0254d8ee6155: Status 404 returned error can't find the container with id fd2c9be6d8f74e9e87d18d4c8b55d482ceea035b00201a74c71f0254d8ee6155 Oct 30 21:53:24.369979 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:24.369928 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:24.370078 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:24.370008 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:24.370078 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:24.370039 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:24.452528 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:24.452497 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" event={"ID":"022d4791-881d-4f32-9531-f0ded16008f3","Type":"ContainerStarted","Data":"5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84"} Oct 30 21:53:24.452656 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:24.452560 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" event={"ID":"022d4791-881d-4f32-9531-f0ded16008f3","Type":"ContainerStarted","Data":"fd2c9be6d8f74e9e87d18d4c8b55d482ceea035b00201a74c71f0254d8ee6155"} Oct 30 21:53:24.576343 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:24.576289 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" podStartSLOduration=1.576277445 podStartE2EDuration="1.576277445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:24.575785611 +0000 UTC m=+661.948412371" watchObservedRunningTime="2026-10-30 21:53:24.576277445 +0000 UTC m=+661.948904190" Oct 30 21:53:24.905615 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:24.905581 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:25.909373 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:25.909343 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:26.464511 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:26.464477 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:26.465197 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:26.465177 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:33.198480 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.198437 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl"] Oct 30 21:53:33.203868 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.203845 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.257755 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.257730 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-282df361-openshift-default-dockercfg-mnq44\"" Oct 30 21:53:33.277491 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.277466 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-istio-token\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.277594 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.277499 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgtjp\" (UniqueName: \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-kube-api-access-xgtjp\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.277594 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.277517 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-credential-socket\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.277594 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.277541 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-data\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.277703 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.277669 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-certs\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.277740 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.277720 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-socket\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.277775 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.277751 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44663585-04f9-473f-bba5-37c8fbe63094-istiod-ca-cert\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.277829 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.277811 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-envoy\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.277863 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.277853 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44663585-04f9-473f-bba5-37c8fbe63094-istio-podinfo\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.308884 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.308859 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl"] Oct 30 21:53:33.379071 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379034 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-data\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379268 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379116 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-certs\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379268 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379167 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-socket\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379268 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379213 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44663585-04f9-473f-bba5-37c8fbe63094-istiod-ca-cert\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379268 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379241 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-envoy\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379457 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379300 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44663585-04f9-473f-bba5-37c8fbe63094-istio-podinfo\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379457 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379331 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-istio-token\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379457 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379365 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgtjp\" (UniqueName: \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-kube-api-access-xgtjp\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379457 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379396 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-credential-socket\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379660 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379523 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-data\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379660 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379569 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-certs\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379763 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379749 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-credential-socket\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.379999 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.379974 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44663585-04f9-473f-bba5-37c8fbe63094-istiod-ca-cert\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.380112 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.380093 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-socket\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.381474 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.381457 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-envoy\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.381707 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.381689 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44663585-04f9-473f-bba5-37c8fbe63094-istio-podinfo\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.439371 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.439341 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-istio-token\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.439521 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.439502 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgtjp\" (UniqueName: \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-kube-api-access-xgtjp\") pod \"e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.523289 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.523212 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:33.948078 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.948053 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl"] Oct 30 21:53:33.948991 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:53:33.948967 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44663585_04f9_473f_bba5_37c8fbe63094.slice/crio-d7e153ab5531484305380e392de3ecb7e5be4961ec202c71934744cf2a16b121 WatchSource:0}: Error finding container d7e153ab5531484305380e392de3ecb7e5be4961ec202c71934744cf2a16b121: Status 404 returned error can't find the container with id d7e153ab5531484305380e392de3ecb7e5be4961ec202c71934744cf2a16b121 Oct 30 21:53:33.951131 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.951090 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:33.951221 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.951163 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:33.951221 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:33.951197 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:34.508802 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:34.508759 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" event={"ID":"44663585-04f9-473f-bba5-37c8fbe63094","Type":"ContainerStarted","Data":"486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf"} Oct 30 21:53:34.509276 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:34.508838 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" event={"ID":"44663585-04f9-473f-bba5-37c8fbe63094","Type":"ContainerStarted","Data":"d7e153ab5531484305380e392de3ecb7e5be4961ec202c71934744cf2a16b121"} Oct 30 21:53:34.524031 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:34.524008 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:34.626096 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:34.626057 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" podStartSLOduration=1.626046645 podStartE2EDuration="1.626046645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:34.624876139 +0000 UTC m=+671.997502884" watchObservedRunningTime="2026-10-30 21:53:34.626046645 +0000 UTC m=+671.998673390" Oct 30 21:53:35.528399 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:35.528325 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:35.948880 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:35.948850 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk"] Oct 30 21:53:35.952590 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:35.952566 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:53:36.010100 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.010070 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f20e6e5e\"" Oct 30 21:53:36.010237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.010130 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f20e6e5e-dockercfg-7rcvv\"" Oct 30 21:53:36.013210 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.013188 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk"] Oct 30 21:53:36.020169 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.020143 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l5cp\" (UniqueName: \"kubernetes.io/projected/2f4f2df1-a3db-4a62-800f-eed1f591d38d-kube-api-access-8l5cp\") pod \"payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk\" (UID: \"2f4f2df1-a3db-4a62-800f-eed1f591d38d\") " pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:53:36.020334 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.020296 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4f2df1-a3db-4a62-800f-eed1f591d38d-plugins-config-volume\") pod \"payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk\" (UID: \"2f4f2df1-a3db-4a62-800f-eed1f591d38d\") " pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:53:36.120838 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.120812 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4f2df1-a3db-4a62-800f-eed1f591d38d-plugins-config-volume\") pod \"payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk\" (UID: \"2f4f2df1-a3db-4a62-800f-eed1f591d38d\") " pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:53:36.121040 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.120880 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l5cp\" (UniqueName: \"kubernetes.io/projected/2f4f2df1-a3db-4a62-800f-eed1f591d38d-kube-api-access-8l5cp\") pod \"payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk\" (UID: \"2f4f2df1-a3db-4a62-800f-eed1f591d38d\") " pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:53:36.121545 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.121516 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4f2df1-a3db-4a62-800f-eed1f591d38d-plugins-config-volume\") pod \"payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk\" (UID: \"2f4f2df1-a3db-4a62-800f-eed1f591d38d\") " pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:53:36.178197 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.178170 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l5cp\" (UniqueName: \"kubernetes.io/projected/2f4f2df1-a3db-4a62-800f-eed1f591d38d-kube-api-access-8l5cp\") pod \"payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk\" (UID: \"2f4f2df1-a3db-4a62-800f-eed1f591d38d\") " pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:53:36.265990 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.265920 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:53:36.519875 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.519800 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:36.520619 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.520595 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:53:36.643372 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.643348 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk"] Oct 30 21:53:36.645825 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:53:36.645796 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4f2df1_a3db_4a62_800f_eed1f591d38d.slice/crio-f686c501cdd5cea537a47415e632d91baf5f4b3df0647deccc9bc0922d711858 WatchSource:0}: Error finding container f686c501cdd5cea537a47415e632d91baf5f4b3df0647deccc9bc0922d711858: Status 404 returned error can't find the container with id f686c501cdd5cea537a47415e632d91baf5f4b3df0647deccc9bc0922d711858 Oct 30 21:53:36.647852 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:36.647833 4859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:53:37.526025 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:37.525992 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" event={"ID":"2f4f2df1-a3db-4a62-800f-eed1f591d38d","Type":"ContainerStarted","Data":"f686c501cdd5cea537a47415e632d91baf5f4b3df0647deccc9bc0922d711858"} Oct 30 21:53:38.357884 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.357851 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc"] Oct 30 21:53:38.362267 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.362241 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:53:38.419024 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.418972 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6996ffc6\"" Oct 30 21:53:38.444974 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.444151 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f85v\" (UniqueName: \"kubernetes.io/projected/43e01e33-ced2-4539-97bf-b449e7594a2f-kube-api-access-2f85v\") pod \"payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc\" (UID: \"43e01e33-ced2-4539-97bf-b449e7594a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:53:38.444974 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.444288 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e01e33-ced2-4539-97bf-b449e7594a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc\" (UID: \"43e01e33-ced2-4539-97bf-b449e7594a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:53:38.471208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.471177 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc"] Oct 30 21:53:38.503730 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.503702 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8"] Oct 30 21:53:38.508121 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.508096 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:53:38.517884 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.517861 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6996ffc6-dockercfg-sm76j\"" Oct 30 21:53:38.544913 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.544878 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cq5p\" (UniqueName: \"kubernetes.io/projected/f7259349-c80c-4290-acef-5c9129d30d5c-kube-api-access-8cq5p\") pod \"payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8\" (UID: \"f7259349-c80c-4290-acef-5c9129d30d5c\") " pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:53:38.545062 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.544957 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f85v\" (UniqueName: \"kubernetes.io/projected/43e01e33-ced2-4539-97bf-b449e7594a2f-kube-api-access-2f85v\") pod \"payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc\" (UID: \"43e01e33-ced2-4539-97bf-b449e7594a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:53:38.545062 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.545025 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e01e33-ced2-4539-97bf-b449e7594a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc\" (UID: \"43e01e33-ced2-4539-97bf-b449e7594a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:53:38.545185 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.545138 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7259349-c80c-4290-acef-5c9129d30d5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8\" (UID: \"f7259349-c80c-4290-acef-5c9129d30d5c\") " pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:53:38.545725 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.545704 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e01e33-ced2-4539-97bf-b449e7594a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc\" (UID: \"43e01e33-ced2-4539-97bf-b449e7594a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:53:38.564871 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.564811 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8"] Oct 30 21:53:38.602512 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.602467 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f85v\" (UniqueName: \"kubernetes.io/projected/43e01e33-ced2-4539-97bf-b449e7594a2f-kube-api-access-2f85v\") pod \"payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc\" (UID: \"43e01e33-ced2-4539-97bf-b449e7594a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:53:38.648578 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.648478 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7259349-c80c-4290-acef-5c9129d30d5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8\" (UID: \"f7259349-c80c-4290-acef-5c9129d30d5c\") " pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:53:38.648745 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.648600 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cq5p\" (UniqueName: \"kubernetes.io/projected/f7259349-c80c-4290-acef-5c9129d30d5c-kube-api-access-8cq5p\") pod \"payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8\" (UID: \"f7259349-c80c-4290-acef-5c9129d30d5c\") " pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:53:38.653534 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.653492 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7259349-c80c-4290-acef-5c9129d30d5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8\" (UID: \"f7259349-c80c-4290-acef-5c9129d30d5c\") " pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:53:38.680787 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.680756 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:53:38.712712 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.712675 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cq5p\" (UniqueName: \"kubernetes.io/projected/f7259349-c80c-4290-acef-5c9129d30d5c-kube-api-access-8cq5p\") pod \"payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8\" (UID: \"f7259349-c80c-4290-acef-5c9129d30d5c\") " pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:53:38.822777 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:38.822742 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:53:39.067266 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:39.067241 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc"] Oct 30 21:53:39.069101 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:53:39.069078 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e01e33_ced2_4539_97bf_b449e7594a2f.slice/crio-1898c47edb153fd5429f54a93784fb92f0c5286efdbac5af1b6f02a3c7db5b35 WatchSource:0}: Error finding container 1898c47edb153fd5429f54a93784fb92f0c5286efdbac5af1b6f02a3c7db5b35: Status 404 returned error can't find the container with id 1898c47edb153fd5429f54a93784fb92f0c5286efdbac5af1b6f02a3c7db5b35 Oct 30 21:53:39.128654 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:39.128620 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:53:39.235411 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:39.235384 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8"] Oct 30 21:53:39.237075 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:53:39.237048 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7259349_c80c_4290_acef_5c9129d30d5c.slice/crio-e1919a0a30fdca74f153a8d48fed19ea9d6631018c9b86824428eaa82bcef509 WatchSource:0}: Error finding container e1919a0a30fdca74f153a8d48fed19ea9d6631018c9b86824428eaa82bcef509: Status 404 returned error can't find the container with id e1919a0a30fdca74f153a8d48fed19ea9d6631018c9b86824428eaa82bcef509 Oct 30 21:53:39.541210 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:39.541136 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" event={"ID":"f7259349-c80c-4290-acef-5c9129d30d5c","Type":"ContainerStarted","Data":"e1919a0a30fdca74f153a8d48fed19ea9d6631018c9b86824428eaa82bcef509"} Oct 30 21:53:39.542534 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:39.542504 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" event={"ID":"43e01e33-ced2-4539-97bf-b449e7594a2f","Type":"ContainerStarted","Data":"1898c47edb153fd5429f54a93784fb92f0c5286efdbac5af1b6f02a3c7db5b35"} Oct 30 21:53:39.878626 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:53:39.878587 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:53:40.550442 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:40.550352 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" event={"ID":"f7259349-c80c-4290-acef-5c9129d30d5c","Type":"ContainerStarted","Data":"7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc"} Oct 30 21:53:40.550850 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:40.550449 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:53:40.551974 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:40.551932 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" event={"ID":"43e01e33-ced2-4539-97bf-b449e7594a2f","Type":"ContainerStarted","Data":"12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd"} Oct 30 21:53:40.552083 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:40.552000 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:53:40.553480 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:40.553454 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" event={"ID":"2f4f2df1-a3db-4a62-800f-eed1f591d38d","Type":"ContainerStarted","Data":"61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077"} Oct 30 21:53:40.553582 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:40.553507 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:53:40.716557 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:40.716510 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" podStartSLOduration=1.859183628 podStartE2EDuration="2.716498235s" totalImagesPullingTime="857.314607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:40.665848844 +0000 UTC m=+678.038475589" watchObservedRunningTime="2026-10-30 21:53:40.716498235 +0000 UTC m=+678.089125001" Oct 30 21:53:40.784194 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:40.784145 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" podStartSLOduration=2.529273767 podStartE2EDuration="5.784123888s" totalImagesPullingTime="3.254850121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:40.78067645 +0000 UTC m=+678.153303199" watchObservedRunningTime="2026-10-30 21:53:40.784123888 +0000 UTC m=+678.156750633" Oct 30 21:53:40.900683 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:40.900625 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" podStartSLOduration=2.042071368 podStartE2EDuration="2.900609454s" totalImagesPullingTime="858.538086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:40.896843405 +0000 UTC m=+678.269470150" watchObservedRunningTime="2026-10-30 21:53:40.900609454 +0000 UTC m=+678.273236229" Oct 30 21:53:47.999051 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:47.999017 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj"] Oct 30 21:53:48.002557 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.002533 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:53:48.038791 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.038764 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzjhh\" (UniqueName: \"kubernetes.io/projected/eed977cb-d19a-4c42-a2b6-187e57037a80-kube-api-access-tzjhh\") pod \"payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj\" (UID: \"eed977cb-d19a-4c42-a2b6-187e57037a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:53:48.038913 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.038890 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed977cb-d19a-4c42-a2b6-187e57037a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj\" (UID: \"eed977cb-d19a-4c42-a2b6-187e57037a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:53:48.056346 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.056322 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj"] Oct 30 21:53:48.104415 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.104384 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-282df361-dockercfg-l26jx\"" Oct 30 21:53:48.104788 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.104689 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-282df361\"" Oct 30 21:53:48.140478 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.140443 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed977cb-d19a-4c42-a2b6-187e57037a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj\" (UID: \"eed977cb-d19a-4c42-a2b6-187e57037a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:53:48.140666 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.140517 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzjhh\" (UniqueName: \"kubernetes.io/projected/eed977cb-d19a-4c42-a2b6-187e57037a80-kube-api-access-tzjhh\") pod \"payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj\" (UID: \"eed977cb-d19a-4c42-a2b6-187e57037a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:53:48.141175 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.141150 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed977cb-d19a-4c42-a2b6-187e57037a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj\" (UID: \"eed977cb-d19a-4c42-a2b6-187e57037a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:53:48.156293 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.156268 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t"] Oct 30 21:53:48.160123 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.160108 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:53:48.196810 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.196779 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzjhh\" (UniqueName: \"kubernetes.io/projected/eed977cb-d19a-4c42-a2b6-187e57037a80-kube-api-access-tzjhh\") pod \"payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj\" (UID: \"eed977cb-d19a-4c42-a2b6-187e57037a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:53:48.203618 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.203590 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t"] Oct 30 21:53:48.241232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.241208 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66d3da08-3bef-4edb-8dd8-fa181eb448ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t\" (UID: \"66d3da08-3bef-4edb-8dd8-fa181eb448ca\") " pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:53:48.241376 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.241348 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jg6d\" (UniqueName: \"kubernetes.io/projected/66d3da08-3bef-4edb-8dd8-fa181eb448ca-kube-api-access-2jg6d\") pod \"payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t\" (UID: \"66d3da08-3bef-4edb-8dd8-fa181eb448ca\") " pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:53:48.316002 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.315897 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:53:48.342704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.342678 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66d3da08-3bef-4edb-8dd8-fa181eb448ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t\" (UID: \"66d3da08-3bef-4edb-8dd8-fa181eb448ca\") " pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:53:48.342833 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.342806 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jg6d\" (UniqueName: \"kubernetes.io/projected/66d3da08-3bef-4edb-8dd8-fa181eb448ca-kube-api-access-2jg6d\") pod \"payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t\" (UID: \"66d3da08-3bef-4edb-8dd8-fa181eb448ca\") " pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:53:48.343394 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.343373 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66d3da08-3bef-4edb-8dd8-fa181eb448ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t\" (UID: \"66d3da08-3bef-4edb-8dd8-fa181eb448ca\") " pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:53:48.409024 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.403900 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jg6d\" (UniqueName: \"kubernetes.io/projected/66d3da08-3bef-4edb-8dd8-fa181eb448ca-kube-api-access-2jg6d\") pod \"payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t\" (UID: \"66d3da08-3bef-4edb-8dd8-fa181eb448ca\") " pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:53:48.499569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.499505 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:53:48.721284 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.721257 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj"] Oct 30 21:53:48.722851 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:53:48.722825 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed977cb_d19a_4c42_a2b6_187e57037a80.slice/crio-1ffd3768d8db892e7c84c62f4ef94c6aeee942de91c97b999ffc2565778f3598 WatchSource:0}: Error finding container 1ffd3768d8db892e7c84c62f4ef94c6aeee942de91c97b999ffc2565778f3598: Status 404 returned error can't find the container with id 1ffd3768d8db892e7c84c62f4ef94c6aeee942de91c97b999ffc2565778f3598 Oct 30 21:53:48.899609 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:48.899577 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t"] Oct 30 21:53:48.903904 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:53:48.903881 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d3da08_3bef_4edb_8dd8_fa181eb448ca.slice/crio-d375e08763958449cd17b3000e74cbfed27af6c2060ae5b074867814661fe209 WatchSource:0}: Error finding container d375e08763958449cd17b3000e74cbfed27af6c2060ae5b074867814661fe209: Status 404 returned error can't find the container with id d375e08763958449cd17b3000e74cbfed27af6c2060ae5b074867814661fe209 Oct 30 21:53:49.624669 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:49.624626 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" event={"ID":"66d3da08-3bef-4edb-8dd8-fa181eb448ca","Type":"ContainerStarted","Data":"4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2"} Oct 30 21:53:49.625185 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:49.624693 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" event={"ID":"66d3da08-3bef-4edb-8dd8-fa181eb448ca","Type":"ContainerStarted","Data":"d375e08763958449cd17b3000e74cbfed27af6c2060ae5b074867814661fe209"} Oct 30 21:53:49.625185 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:49.624721 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:53:49.626370 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:49.626347 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" event={"ID":"eed977cb-d19a-4c42-a2b6-187e57037a80","Type":"ContainerStarted","Data":"75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c"} Oct 30 21:53:49.626472 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:49.626385 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" event={"ID":"eed977cb-d19a-4c42-a2b6-187e57037a80","Type":"ContainerStarted","Data":"1ffd3768d8db892e7c84c62f4ef94c6aeee942de91c97b999ffc2565778f3598"} Oct 30 21:53:49.626472 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:49.626413 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:53:49.748114 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:49.748067 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" podStartSLOduration=1.748054078 podStartE2EDuration="1.748054078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:49.746466638 +0000 UTC m=+687.119093385" watchObservedRunningTime="2026-10-30 21:53:49.748054078 +0000 UTC m=+687.120680823" Oct 30 21:53:49.883021 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:49.880540 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" podStartSLOduration=2.8805220289999998 podStartE2EDuration="2.880522029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:49.866276853 +0000 UTC m=+687.238903599" watchObservedRunningTime="2026-10-30 21:53:49.880522029 +0000 UTC m=+687.253148776" Oct 30 21:53:50.586972 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.586022 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g"] Oct 30 21:53:50.592442 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.592412 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.648625 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.648590 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g"] Oct 30 21:53:50.691971 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.689883 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istiod-ca-cert\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.691971 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.689962 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-envoy\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.691971 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.690003 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-data\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.691971 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.690037 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-podinfo\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.691971 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.690072 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-socket\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.691971 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.690148 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-certs\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.691971 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.690589 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-credential-socket\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.691971 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.690685 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-token\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.691971 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.690767 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnx8k\" (UniqueName: \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-kube-api-access-nnx8k\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.691971 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.690781 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c678f1ee-openshift-default-dockercfg-h546g\"" Oct 30 21:53:50.791533 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.791501 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-socket\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.791533 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.791539 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-certs\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.791788 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.791567 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-credential-socket\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.791788 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.791606 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-token\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.791788 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.791647 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnx8k\" (UniqueName: \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-kube-api-access-nnx8k\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.791788 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.791681 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istiod-ca-cert\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.792026 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.791927 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-envoy\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.792434 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.792419 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-data\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.792474 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.791971 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-credential-socket\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.792474 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.792465 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-podinfo\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.792557 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.792070 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-certs\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.792557 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.792050 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-socket\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.792557 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.792365 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istiod-ca-cert\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.792822 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.792799 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-data\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.793980 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.793940 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-envoy\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.794514 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.794496 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-podinfo\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.852215 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.852185 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-token\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.852434 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.852412 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnx8k\" (UniqueName: \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-kube-api-access-nnx8k\") pod \"e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:50.914550 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:50.914519 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:51.388784 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.388744 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g"] Oct 30 21:53:51.411640 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:53:51.411602 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b4dfab_3cc0_4287_a018_dbf68d67f9b0.slice/crio-c79ba669667f433c3b912e1507fe477eed4542b2f08f42b60a11b99ee3de24bf WatchSource:0}: Error finding container c79ba669667f433c3b912e1507fe477eed4542b2f08f42b60a11b99ee3de24bf: Status 404 returned error can't find the container with id c79ba669667f433c3b912e1507fe477eed4542b2f08f42b60a11b99ee3de24bf Oct 30 21:53:51.413861 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.413826 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:51.413963 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.413908 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:51.414032 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.413981 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:51.587872 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.587150 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:53:51.588709 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.588500 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:53:51.588709 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.588567 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:53:51.647096 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.647023 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" event={"ID":"43b4dfab-3cc0-4287-a018-dbf68d67f9b0","Type":"ContainerStarted","Data":"13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84"} Oct 30 21:53:51.647096 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.647076 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" event={"ID":"43b4dfab-3cc0-4287-a018-dbf68d67f9b0","Type":"ContainerStarted","Data":"c79ba669667f433c3b912e1507fe477eed4542b2f08f42b60a11b99ee3de24bf"} Oct 30 21:53:51.914743 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.914668 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:51.954850 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:51.954803 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" podStartSLOduration=1.9547907329999998 podStartE2EDuration="1.954790733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:51.832073106 +0000 UTC m=+689.204699851" watchObservedRunningTime="2026-10-30 21:53:51.954790733 +0000 UTC m=+689.327417482" Oct 30 21:53:52.919912 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:52.919873 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:53.658543 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:53.658508 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:53.659772 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:53.659749 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:53:56.466315 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:56.466282 4859 patch_prober.go:29] interesting pod/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:56.466843 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:56.466341 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 21:53:56.677650 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:56.677619 4859 generic.go:356] "Generic (PLEG): container finished" podID="022d4791-881d-4f32-9531-f0ded16008f3" containerID="5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84" exitCode=0 Oct 30 21:53:56.677835 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:56.677694 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" event={"ID":"022d4791-881d-4f32-9531-f0ded16008f3","Type":"ContainerDied","Data":"5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84"} Oct 30 21:53:56.678130 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:56.678112 4859 scope.go:122] "RemoveContainer" containerID="5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84" Oct 30 21:53:56.679302 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:56.679277 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:56.679368 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:56.679343 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:56.679417 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:56.679374 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:53:56.993680 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:56.993583 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:53:57.705919 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:57.705887 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" event={"ID":"022d4791-881d-4f32-9531-f0ded16008f3","Type":"ContainerStarted","Data":"d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b"} Oct 30 21:53:57.905689 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:57.905659 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:53:57.905885 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:57.905864 4859 patch_prober.go:29] interesting pod/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:57.905931 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:57.905911 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 21:53:58.905540 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:58.905506 4859 patch_prober.go:29] interesting pod/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:58.905911 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:58.905551 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 21:53:59.905790 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:59.905744 4859 patch_prober.go:29] interesting pod/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:59.906284 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:53:59.905811 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 21:54:00.393456 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.393420 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd"] Oct 30 21:54:00.397655 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.397637 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:54:00.507968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.507243 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c678f1ee-dockercfg-5hgzk\"" Oct 30 21:54:00.507968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.507423 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c678f1ee\"" Oct 30 21:54:00.521792 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.521517 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mgr6\" (UniqueName: \"kubernetes.io/projected/5a604d62-838f-4207-9879-ab3688c30412-kube-api-access-8mgr6\") pod \"payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd\" (UID: \"5a604d62-838f-4207-9879-ab3688c30412\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:54:00.521792 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.521639 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a604d62-838f-4207-9879-ab3688c30412-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd\" (UID: \"5a604d62-838f-4207-9879-ab3688c30412\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:54:00.526179 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.526002 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd"] Oct 30 21:54:00.535280 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.532446 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p"] Oct 30 21:54:00.536776 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.536299 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:54:00.597076 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.597045 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p"] Oct 30 21:54:00.623321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.623283 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prr7f\" (UniqueName: \"kubernetes.io/projected/9417396d-56c7-45a6-b419-1664028587ca-kube-api-access-prr7f\") pod \"payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p\" (UID: \"9417396d-56c7-45a6-b419-1664028587ca\") " pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:54:00.623494 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.623350 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a604d62-838f-4207-9879-ab3688c30412-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd\" (UID: \"5a604d62-838f-4207-9879-ab3688c30412\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:54:00.623764 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.623738 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9417396d-56c7-45a6-b419-1664028587ca-plugins-config-volume\") pod \"payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p\" (UID: \"9417396d-56c7-45a6-b419-1664028587ca\") " pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:54:00.623939 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.623918 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mgr6\" (UniqueName: \"kubernetes.io/projected/5a604d62-838f-4207-9879-ab3688c30412-kube-api-access-8mgr6\") pod \"payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd\" (UID: \"5a604d62-838f-4207-9879-ab3688c30412\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:54:00.624236 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.624215 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a604d62-838f-4207-9879-ab3688c30412-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd\" (UID: \"5a604d62-838f-4207-9879-ab3688c30412\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:54:00.643017 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.642994 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:54:00.643141 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.643122 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:54:00.698853 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.698789 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mgr6\" (UniqueName: \"kubernetes.io/projected/5a604d62-838f-4207-9879-ab3688c30412-kube-api-access-8mgr6\") pod \"payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd\" (UID: \"5a604d62-838f-4207-9879-ab3688c30412\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:54:00.709312 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.709287 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:54:00.724828 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.724805 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9417396d-56c7-45a6-b419-1664028587ca-plugins-config-volume\") pod \"payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p\" (UID: \"9417396d-56c7-45a6-b419-1664028587ca\") " pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:54:00.724938 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.724881 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prr7f\" (UniqueName: \"kubernetes.io/projected/9417396d-56c7-45a6-b419-1664028587ca-kube-api-access-prr7f\") pod \"payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p\" (UID: \"9417396d-56c7-45a6-b419-1664028587ca\") " pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:54:00.725864 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.725824 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9417396d-56c7-45a6-b419-1664028587ca-plugins-config-volume\") pod \"payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p\" (UID: \"9417396d-56c7-45a6-b419-1664028587ca\") " pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:54:00.798246 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.794614 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prr7f\" (UniqueName: \"kubernetes.io/projected/9417396d-56c7-45a6-b419-1664028587ca-kube-api-access-prr7f\") pod \"payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p\" (UID: \"9417396d-56c7-45a6-b419-1664028587ca\") " pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:54:00.855209 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.855167 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:54:00.905732 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.905705 4859 patch_prober.go:29] interesting pod/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:54:00.905847 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:00.905755 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 21:54:01.118133 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.118109 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd"] Oct 30 21:54:01.119208 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:54:01.119181 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a604d62_838f_4207_9879_ab3688c30412.slice/crio-70b6fd33f65c75206023db0d2a84fe16c902ffcaeb42fd2a4d20a8fed8dbfabb WatchSource:0}: Error finding container 70b6fd33f65c75206023db0d2a84fe16c902ffcaeb42fd2a4d20a8fed8dbfabb: Status 404 returned error can't find the container with id 70b6fd33f65c75206023db0d2a84fe16c902ffcaeb42fd2a4d20a8fed8dbfabb Oct 30 21:54:01.279151 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.279113 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p"] Oct 30 21:54:01.732235 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.732184 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" event={"ID":"9417396d-56c7-45a6-b419-1664028587ca","Type":"ContainerStarted","Data":"9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957"} Oct 30 21:54:01.732563 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.732541 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:54:01.732718 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.732703 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" event={"ID":"9417396d-56c7-45a6-b419-1664028587ca","Type":"ContainerStarted","Data":"50165e7937d570a4d9945059db6a1d9337e79162328fae7bf5e76842fe9a0138"} Oct 30 21:54:01.733678 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.733644 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" event={"ID":"5a604d62-838f-4207-9879-ab3688c30412","Type":"ContainerStarted","Data":"b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d"} Oct 30 21:54:01.733773 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.733700 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" event={"ID":"5a604d62-838f-4207-9879-ab3688c30412","Type":"ContainerStarted","Data":"70b6fd33f65c75206023db0d2a84fe16c902ffcaeb42fd2a4d20a8fed8dbfabb"} Oct 30 21:54:01.733973 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.733933 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:54:01.848886 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.848838 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" podStartSLOduration=1.8488245970000001 podStartE2EDuration="1.848824597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:01.847581765 +0000 UTC m=+699.220208512" watchObservedRunningTime="2026-10-30 21:54:01.848824597 +0000 UTC m=+699.221451341" Oct 30 21:54:01.905517 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.905491 4859 patch_prober.go:29] interesting pod/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:54:01.905719 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.905696 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 21:54:01.982539 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:01.982420 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" podStartSLOduration=1.982400029 podStartE2EDuration="1.982400029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:01.972718388 +0000 UTC m=+699.345345150" watchObservedRunningTime="2026-10-30 21:54:01.982400029 +0000 UTC m=+699.355026774" Oct 30 21:54:02.909679 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:02.909647 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:54:02.910085 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:02.910063 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:54:02.910658 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:02.910639 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:54:06.379291 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:06.379260 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:54:09.911185 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:54:09.911145 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:54:12.742864 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:12.742834 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:54:13.748883 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:13.748855 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:54:15.842443 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:15.842410 4859 generic.go:356] "Generic (PLEG): container finished" podID="44663585-04f9-473f-bba5-37c8fbe63094" containerID="486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf" exitCode=0 Oct 30 21:54:15.842851 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:15.842499 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" event={"ID":"44663585-04f9-473f-bba5-37c8fbe63094","Type":"ContainerDied","Data":"486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf"} Oct 30 21:54:15.842932 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:15.842893 4859 scope.go:122] "RemoveContainer" containerID="486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf" Oct 30 21:54:15.844270 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:15.844234 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:15.844341 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:15.844304 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:15.844341 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:15.844334 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:16.850520 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:16.850486 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" event={"ID":"44663585-04f9-473f-bba5-37c8fbe63094","Type":"ContainerStarted","Data":"11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4"} Oct 30 21:54:17.523404 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:17.523354 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:54:17.524588 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:17.524565 4859 patch_prober.go:29] interesting pod/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:54:17.524656 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:17.524613 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:54:18.523668 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:18.523637 4859 patch_prober.go:29] interesting pod/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:54:18.524094 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:18.523698 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:54:19.523538 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:19.523504 4859 patch_prober.go:29] interesting pod/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:54:19.523791 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:19.523557 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:54:20.523579 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:20.523538 4859 patch_prober.go:29] interesting pod/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:54:20.523745 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:20.523608 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:54:21.520062 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:21.520025 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:54:21.523781 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:21.523754 4859 patch_prober.go:29] interesting pod/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:54:21.523916 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:21.523805 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:54:22.528011 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:22.527979 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:54:22.528901 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:22.528886 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:54:25.624519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.624483 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72"] Oct 30 21:54:25.632218 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.632191 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:25.676823 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.676795 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:25.677001 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.676978 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:25.677080 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.677048 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwsb\" (UniqueName: \"kubernetes.io/projected/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-kube-api-access-5wwsb\") pod \"maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:25.688731 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.688708 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:25.688731 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.688721 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\"" Oct 30 21:54:25.690435 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.690413 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72"] Oct 30 21:54:25.777579 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.777554 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:25.777711 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.777593 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwsb\" (UniqueName: \"kubernetes.io/projected/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-kube-api-access-5wwsb\") pod \"maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:25.777711 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.777635 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:25.779716 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.779698 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:25.780376 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.780361 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:25.780470 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.780457 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:25.788969 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.788933 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:25.834432 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.834412 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwsb\" (UniqueName: \"kubernetes.io/projected/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-kube-api-access-5wwsb\") pod \"maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:25.945462 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:25.945385 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:26.329282 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:54:26.329249 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9708a9f_23d1_4164_8d2c_78b74c4fd25c.slice/crio-297aeb3b87fcede83978770e82cb0035e5a9d231448458b5fa9b2f902171a037 WatchSource:0}: Error finding container 297aeb3b87fcede83978770e82cb0035e5a9d231448458b5fa9b2f902171a037: Status 404 returned error can't find the container with id 297aeb3b87fcede83978770e82cb0035e5a9d231448458b5fa9b2f902171a037 Oct 30 21:54:26.330130 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:26.330072 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72"] Oct 30 21:54:26.952888 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:26.952855 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" event={"ID":"a9708a9f-23d1-4164-8d2c-78b74c4fd25c","Type":"ContainerStarted","Data":"297aeb3b87fcede83978770e82cb0035e5a9d231448458b5fa9b2f902171a037"} Oct 30 21:54:29.505800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:29.505763 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:54:30.998248 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:30.998206 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" event={"ID":"a9708a9f-23d1-4164-8d2c-78b74c4fd25c","Type":"ContainerStarted","Data":"2b1b355b923aac167944b3210b71b55261f92a18a27e1b25adbec0d2a212c1b7"} Oct 30 21:54:31.121046 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:31.120985 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" podStartSLOduration=2.421827654 podStartE2EDuration="6.120966866s" totalImagesPullingTime="3.699139212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:31.117570092 +0000 UTC m=+728.490196837" watchObservedRunningTime="2026-10-30 21:54:31.120966866 +0000 UTC m=+728.493593610" Oct 30 21:54:32.005678 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:32.005648 4859 generic.go:356] "Generic (PLEG): container finished" podID="a9708a9f-23d1-4164-8d2c-78b74c4fd25c" containerID="2b1b355b923aac167944b3210b71b55261f92a18a27e1b25adbec0d2a212c1b7" exitCode=0 Oct 30 21:54:32.006086 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:32.005736 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" event={"ID":"a9708a9f-23d1-4164-8d2c-78b74c4fd25c","Type":"ContainerDied","Data":"2b1b355b923aac167944b3210b71b55261f92a18a27e1b25adbec0d2a212c1b7"} Oct 30 21:54:33.192432 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:33.192408 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:33.257880 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:33.257855 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-openshift-ca\") pod \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " Oct 30 21:54:33.258027 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:33.257898 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-xks-ca\") pod \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " Oct 30 21:54:33.258027 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:33.257977 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-kube-api-access-5wwsb\" (UniqueName: \"kubernetes.io/projected/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-kube-api-access-5wwsb\") pod \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\" (UID: \"a9708a9f-23d1-4164-8d2c-78b74c4fd25c\") " Oct 30 21:54:33.258267 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:33.258241 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-openshift-ca" pod "a9708a9f-23d1-4164-8d2c-78b74c4fd25c" (UID: "a9708a9f-23d1-4164-8d2c-78b74c4fd25c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:33.260114 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:33.260083 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-xks-ca" pod "a9708a9f-23d1-4164-8d2c-78b74c4fd25c" (UID: "a9708a9f-23d1-4164-8d2c-78b74c4fd25c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:33.260214 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:33.260144 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-kube-api-access-5wwsb" pod "a9708a9f-23d1-4164-8d2c-78b74c4fd25c" (UID: "a9708a9f-23d1-4164-8d2c-78b74c4fd25c"). InnerVolumeSpecName "kube-api-access-5wwsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:33.358901 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:33.358872 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wwsb\" (UniqueName: \"kubernetes.io/projected/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-kube-api-access-5wwsb\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:33.358901 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:33.358897 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-openshift-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:33.359085 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:33.358908 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9708a9f-23d1-4164-8d2c-78b74c4fd25c-maas-api-xks-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:34.019771 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:34.019745 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" Oct 30 21:54:34.019960 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:34.019739 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72" event={"ID":"a9708a9f-23d1-4164-8d2c-78b74c4fd25c","Type":"ContainerDied","Data":"297aeb3b87fcede83978770e82cb0035e5a9d231448458b5fa9b2f902171a037"} Oct 30 21:54:34.019960 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:34.019865 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="297aeb3b87fcede83978770e82cb0035e5a9d231448458b5fa9b2f902171a037" Oct 30 21:54:36.068558 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:36.068527 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk"] Oct 30 21:54:38.577711 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:38.577674 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8"] Oct 30 21:54:38.660355 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:38.660330 4859 patch_prober.go:29] interesting pod/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:38.660476 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:38.660381 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:54:38.686148 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:38.686120 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc"] Oct 30 21:54:39.053125 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:39.053095 4859 generic.go:356] "Generic (PLEG): container finished" podID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerID="13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84" exitCode=0 Oct 30 21:54:39.053282 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:39.053146 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" event={"ID":"43b4dfab-3cc0-4287-a018-dbf68d67f9b0","Type":"ContainerDied","Data":"13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84"} Oct 30 21:54:39.053522 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:39.053507 4859 scope.go:122] "RemoveContainer" containerID="13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84" Oct 30 21:54:39.054737 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:39.054709 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:39.054800 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:39.054788 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:39.054835 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:39.054822 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:39.895916 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:54:39.895879 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:54:40.098975 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:40.097996 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" event={"ID":"43b4dfab-3cc0-4287-a018-dbf68d67f9b0","Type":"ContainerStarted","Data":"f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0"} Oct 30 21:54:40.914754 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:40.914721 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:54:40.914754 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:40.914751 4859 patch_prober.go:29] interesting pod/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:40.915199 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:40.914802 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:54:41.069412 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:41.069365 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" podUID="2f4f2df1-a3db-4a62-800f-eed1f591d38d" containerName="payload-processing" containerID="cri-o://61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077" gracePeriod=25 Oct 30 21:54:41.378927 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:41.378585 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:54:41.423481 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:41.423451 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4f2df1-a3db-4a62-800f-eed1f591d38d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4f2df1-a3db-4a62-800f-eed1f591d38d-plugins-config-volume\") pod \"2f4f2df1-a3db-4a62-800f-eed1f591d38d\" (UID: \"2f4f2df1-a3db-4a62-800f-eed1f591d38d\") " Oct 30 21:54:41.423626 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:41.423541 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4f2df1-a3db-4a62-800f-eed1f591d38d-kube-api-access-8l5cp\" (UniqueName: \"kubernetes.io/projected/2f4f2df1-a3db-4a62-800f-eed1f591d38d-kube-api-access-8l5cp\") pod \"2f4f2df1-a3db-4a62-800f-eed1f591d38d\" (UID: \"2f4f2df1-a3db-4a62-800f-eed1f591d38d\") " Oct 30 21:54:41.423864 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:41.423839 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4f2df1-a3db-4a62-800f-eed1f591d38d-plugins-config-volume" pod "2f4f2df1-a3db-4a62-800f-eed1f591d38d" (UID: "2f4f2df1-a3db-4a62-800f-eed1f591d38d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:41.423961 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:41.423929 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4f2df1-a3db-4a62-800f-eed1f591d38d-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.425751 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:41.425723 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4f2df1-a3db-4a62-800f-eed1f591d38d-kube-api-access-8l5cp" pod "2f4f2df1-a3db-4a62-800f-eed1f591d38d" (UID: "2f4f2df1-a3db-4a62-800f-eed1f591d38d"). InnerVolumeSpecName "kube-api-access-8l5cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:41.525051 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:41.524968 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l5cp\" (UniqueName: \"kubernetes.io/projected/2f4f2df1-a3db-4a62-800f-eed1f591d38d-kube-api-access-8l5cp\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.914840 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:41.914809 4859 patch_prober.go:29] interesting pod/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:41.915280 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:41.914861 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:54:42.120601 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.120568 4859 generic.go:356] "Generic (PLEG): container finished" podID="2f4f2df1-a3db-4a62-800f-eed1f591d38d" containerID="61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077" exitCode=0 Oct 30 21:54:42.120784 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.120631 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" Oct 30 21:54:42.120784 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.120642 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" event={"ID":"2f4f2df1-a3db-4a62-800f-eed1f591d38d","Type":"ContainerDied","Data":"61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077"} Oct 30 21:54:42.120784 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.120702 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk" event={"ID":"2f4f2df1-a3db-4a62-800f-eed1f591d38d","Type":"ContainerDied","Data":"f686c501cdd5cea537a47415e632d91baf5f4b3df0647deccc9bc0922d711858"} Oct 30 21:54:42.120784 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.120727 4859 scope.go:122] "RemoveContainer" containerID="61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077" Oct 30 21:54:42.143071 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.143049 4859 scope.go:122] "RemoveContainer" containerID="61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077" Oct 30 21:54:42.143372 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:54:42.143347 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077\": container with ID starting with 61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077 not found: ID does not exist" containerID="61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077" Oct 30 21:54:42.143446 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.143380 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077"} err="failed to get container status \"61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077\": rpc error: code = NotFound desc = could not find container \"61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077\": container with ID starting with 61e38d38a524c652a5b134ec165b86b8cfda244bfae11e37740779a0d8545077 not found: ID does not exist" Oct 30 21:54:42.148549 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:54:42.148528 4859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4f2df1_a3db_4a62_800f_eed1f591d38d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4f2df1_a3db_4a62_800f_eed1f591d38d.slice/crio-f686c501cdd5cea537a47415e632d91baf5f4b3df0647deccc9bc0922d711858\": RecentStats: unable to find data in memory cache]" Oct 30 21:54:42.148549 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:54:42.148541 4859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4f2df1_a3db_4a62_800f_eed1f591d38d.slice\": RecentStats: unable to find data in memory cache]" Oct 30 21:54:42.289077 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.288998 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk"] Oct 30 21:54:42.296910 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.296883 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f20e6e5e-7df5c7ff47-29bmk"] Oct 30 21:54:42.741361 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.741332 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4f2df1-a3db-4a62-800f-eed1f591d38d" path="/var/lib/kubelet/pods/2f4f2df1-a3db-4a62-800f-eed1f591d38d/volumes" Oct 30 21:54:42.914820 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.914788 4859 patch_prober.go:29] interesting pod/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:42.915021 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:42.914839 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:54:43.584983 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:43.580999 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" podUID="f7259349-c80c-4290-acef-5c9129d30d5c" containerName="payload-processing" containerID="cri-o://7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc" gracePeriod=25 Oct 30 21:54:43.690764 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:43.688011 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" podUID="43e01e33-ced2-4539-97bf-b449e7594a2f" containerName="payload-pre-processing" containerID="cri-o://12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd" gracePeriod=25 Oct 30 21:54:43.918336 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:43.917857 4859 patch_prober.go:29] interesting pod/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:43.918336 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:43.917911 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:54:43.926714 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:43.925887 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:54:43.948592 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:43.948558 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7259349-c80c-4290-acef-5c9129d30d5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7259349-c80c-4290-acef-5c9129d30d5c-plugins-config-volume\") pod \"f7259349-c80c-4290-acef-5c9129d30d5c\" (UID: \"f7259349-c80c-4290-acef-5c9129d30d5c\") " Oct 30 21:54:43.948732 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:43.948675 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7259349-c80c-4290-acef-5c9129d30d5c-kube-api-access-8cq5p\" (UniqueName: \"kubernetes.io/projected/f7259349-c80c-4290-acef-5c9129d30d5c-kube-api-access-8cq5p\") pod \"f7259349-c80c-4290-acef-5c9129d30d5c\" (UID: \"f7259349-c80c-4290-acef-5c9129d30d5c\") " Oct 30 21:54:43.949665 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:43.949634 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7259349-c80c-4290-acef-5c9129d30d5c-plugins-config-volume" pod "f7259349-c80c-4290-acef-5c9129d30d5c" (UID: "f7259349-c80c-4290-acef-5c9129d30d5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:43.952662 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:43.952628 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7259349-c80c-4290-acef-5c9129d30d5c-kube-api-access-8cq5p" pod "f7259349-c80c-4290-acef-5c9129d30d5c" (UID: "f7259349-c80c-4290-acef-5c9129d30d5c"). InnerVolumeSpecName "kube-api-access-8cq5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:44.012365 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.012344 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:54:44.049970 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.049835 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7259349-c80c-4290-acef-5c9129d30d5c-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:44.049970 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.049863 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cq5p\" (UniqueName: \"kubernetes.io/projected/f7259349-c80c-4290-acef-5c9129d30d5c-kube-api-access-8cq5p\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:44.144145 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.144064 4859 generic.go:356] "Generic (PLEG): container finished" podID="43e01e33-ced2-4539-97bf-b449e7594a2f" containerID="12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd" exitCode=0 Oct 30 21:54:44.144313 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.144145 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" event={"ID":"43e01e33-ced2-4539-97bf-b449e7594a2f","Type":"ContainerDied","Data":"12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd"} Oct 30 21:54:44.144443 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.144425 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" Oct 30 21:54:44.144673 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.144651 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc" event={"ID":"43e01e33-ced2-4539-97bf-b449e7594a2f","Type":"ContainerDied","Data":"1898c47edb153fd5429f54a93784fb92f0c5286efdbac5af1b6f02a3c7db5b35"} Oct 30 21:54:44.144771 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.144696 4859 scope.go:122] "RemoveContainer" containerID="12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd" Oct 30 21:54:44.147324 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.147299 4859 generic.go:356] "Generic (PLEG): container finished" podID="f7259349-c80c-4290-acef-5c9129d30d5c" containerID="7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc" exitCode=0 Oct 30 21:54:44.147470 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.147396 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" event={"ID":"f7259349-c80c-4290-acef-5c9129d30d5c","Type":"ContainerDied","Data":"7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc"} Oct 30 21:54:44.147470 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.147432 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" event={"ID":"f7259349-c80c-4290-acef-5c9129d30d5c","Type":"ContainerDied","Data":"e1919a0a30fdca74f153a8d48fed19ea9d6631018c9b86824428eaa82bcef509"} Oct 30 21:54:44.147618 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.147487 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8" Oct 30 21:54:44.150393 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.150367 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e01e33-ced2-4539-97bf-b449e7594a2f-kube-api-access-2f85v\" (UniqueName: \"kubernetes.io/projected/43e01e33-ced2-4539-97bf-b449e7594a2f-kube-api-access-2f85v\") pod \"43e01e33-ced2-4539-97bf-b449e7594a2f\" (UID: \"43e01e33-ced2-4539-97bf-b449e7594a2f\") " Oct 30 21:54:44.150482 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.150407 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e01e33-ced2-4539-97bf-b449e7594a2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e01e33-ced2-4539-97bf-b449e7594a2f-plugins-config-volume\") pod \"43e01e33-ced2-4539-97bf-b449e7594a2f\" (UID: \"43e01e33-ced2-4539-97bf-b449e7594a2f\") " Oct 30 21:54:44.152629 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.152595 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e01e33-ced2-4539-97bf-b449e7594a2f-plugins-config-volume" pod "43e01e33-ced2-4539-97bf-b449e7594a2f" (UID: "43e01e33-ced2-4539-97bf-b449e7594a2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:44.152809 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.152784 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e01e33-ced2-4539-97bf-b449e7594a2f-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:44.185182 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.175139 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e01e33-ced2-4539-97bf-b449e7594a2f-kube-api-access-2f85v" pod "43e01e33-ced2-4539-97bf-b449e7594a2f" (UID: "43e01e33-ced2-4539-97bf-b449e7594a2f"). InnerVolumeSpecName "kube-api-access-2f85v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:44.203278 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.196894 4859 scope.go:122] "RemoveContainer" containerID="12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd" Oct 30 21:54:44.203278 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:54:44.197566 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd\": container with ID starting with 12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd not found: ID does not exist" containerID="12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd" Oct 30 21:54:44.203278 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.197633 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd"} err="failed to get container status \"12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd\": rpc error: code = NotFound desc = could not find container \"12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd\": container with ID starting with 12546455b4226acc794975ae1aa9340221032deca4288f505161e86ed08e14dd not found: ID does not exist" Oct 30 21:54:44.203278 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.197661 4859 scope.go:122] "RemoveContainer" containerID="7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc" Oct 30 21:54:44.217635 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.217611 4859 scope.go:122] "RemoveContainer" containerID="7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc" Oct 30 21:54:44.218055 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:54:44.218030 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc\": container with ID starting with 7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc not found: ID does not exist" containerID="7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc" Oct 30 21:54:44.218161 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.218061 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc"} err="failed to get container status \"7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc\": rpc error: code = NotFound desc = could not find container \"7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc\": container with ID starting with 7e394fdfdac2fe3677abae198b94dcdd62dc2a239d4094caaff5c5a1b3a4d4dc not found: ID does not exist" Oct 30 21:54:44.254372 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.254336 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f85v\" (UniqueName: \"kubernetes.io/projected/43e01e33-ced2-4539-97bf-b449e7594a2f-kube-api-access-2f85v\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:44.291969 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.291914 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8"] Oct 30 21:54:44.294533 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.294507 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6996ffc6-5cbc494b49-vjdt8"] Oct 30 21:54:44.596045 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.596005 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc"] Oct 30 21:54:44.607231 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.606295 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6996ffc6-546d54f6dc-l5dkc"] Oct 30 21:54:44.744562 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.744484 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e01e33-ced2-4539-97bf-b449e7594a2f" path="/var/lib/kubelet/pods/43e01e33-ced2-4539-97bf-b449e7594a2f/volumes" Oct 30 21:54:44.744868 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.744849 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7259349-c80c-4290-acef-5c9129d30d5c" path="/var/lib/kubelet/pods/f7259349-c80c-4290-acef-5c9129d30d5c/volumes" Oct 30 21:54:44.866449 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.866413 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt"] Oct 30 21:54:44.867255 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867234 4859 cpu_manager.go:414] "containerMap: removing container" podUID="a9708a9f-23d1-4164-8d2c-78b74c4fd25c" containerName="revoke-keys" Oct 30 21:54:44.867381 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867370 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9708a9f-23d1-4164-8d2c-78b74c4fd25c" containerName="revoke-keys" Oct 30 21:54:44.867450 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867443 4859 cpu_manager.go:414] "containerMap: removing container" podUID="43e01e33-ced2-4539-97bf-b449e7594a2f" containerName="payload-pre-processing" Oct 30 21:54:44.867498 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867491 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e01e33-ced2-4539-97bf-b449e7594a2f" containerName="payload-pre-processing" Oct 30 21:54:44.867551 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867545 4859 cpu_manager.go:414] "containerMap: removing container" podUID="2f4f2df1-a3db-4a62-800f-eed1f591d38d" containerName="payload-processing" Oct 30 21:54:44.867595 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867589 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4f2df1-a3db-4a62-800f-eed1f591d38d" containerName="payload-processing" Oct 30 21:54:44.867647 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867641 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f7259349-c80c-4290-acef-5c9129d30d5c" containerName="payload-processing" Oct 30 21:54:44.867694 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867688 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7259349-c80c-4290-acef-5c9129d30d5c" containerName="payload-processing" Oct 30 21:54:44.867825 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867817 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9708a9f-23d1-4164-8d2c-78b74c4fd25c" containerName="revoke-keys" Oct 30 21:54:44.867877 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867871 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e01e33-ced2-4539-97bf-b449e7594a2f" containerName="payload-pre-processing" Oct 30 21:54:44.867924 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867918 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4f2df1-a3db-4a62-800f-eed1f591d38d" containerName="payload-processing" Oct 30 21:54:44.868003 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.867982 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7259349-c80c-4290-acef-5c9129d30d5c" containerName="payload-processing" Oct 30 21:54:44.870663 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.870646 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 21:54:44.915286 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.915242 4859 patch_prober.go:29] interesting pod/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:44.915472 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.915327 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:54:44.926406 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.926374 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt"] Oct 30 21:54:44.926802 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.926661 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-e90e35\"" Oct 30 21:54:44.926802 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.926740 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-e90e35-dockercfg-8kwsr\"" Oct 30 21:54:44.960242 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.960214 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d7d334-d9ba-4857-9161-3c5fc602194b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt\" (UID: \"39d7d334-d9ba-4857-9161-3c5fc602194b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 21:54:44.960368 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:44.960247 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfl42\" (UniqueName: \"kubernetes.io/projected/39d7d334-d9ba-4857-9161-3c5fc602194b-kube-api-access-wfl42\") pod \"payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt\" (UID: \"39d7d334-d9ba-4857-9161-3c5fc602194b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 21:54:45.061583 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.061483 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d7d334-d9ba-4857-9161-3c5fc602194b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt\" (UID: \"39d7d334-d9ba-4857-9161-3c5fc602194b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 21:54:45.061583 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.061528 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfl42\" (UniqueName: \"kubernetes.io/projected/39d7d334-d9ba-4857-9161-3c5fc602194b-kube-api-access-wfl42\") pod \"payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt\" (UID: \"39d7d334-d9ba-4857-9161-3c5fc602194b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 21:54:45.062200 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.062177 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d7d334-d9ba-4857-9161-3c5fc602194b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt\" (UID: \"39d7d334-d9ba-4857-9161-3c5fc602194b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 21:54:45.118096 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.118065 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfl42\" (UniqueName: \"kubernetes.io/projected/39d7d334-d9ba-4857-9161-3c5fc602194b-kube-api-access-wfl42\") pod \"payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt\" (UID: \"39d7d334-d9ba-4857-9161-3c5fc602194b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 21:54:45.182511 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.182472 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 21:54:45.333153 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.333091 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72"] Oct 30 21:54:45.337006 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.336983 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6996ffc6-q8j72"] Oct 30 21:54:45.562781 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.562753 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt"] Oct 30 21:54:45.564167 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:54:45.564142 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d7d334_d9ba_4857_9161_3c5fc602194b.slice/crio-09f05e0d2c776ed4ce27fb006ab062bd43cd29640f349e611011b605fa2ece2a WatchSource:0}: Error finding container 09f05e0d2c776ed4ce27fb006ab062bd43cd29640f349e611011b605fa2ece2a: Status 404 returned error can't find the container with id 09f05e0d2c776ed4ce27fb006ab062bd43cd29640f349e611011b605fa2ece2a Oct 30 21:54:45.919274 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.919197 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:54:45.919610 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.919593 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:54:45.920324 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:45.920303 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:54:46.192037 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:46.185928 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" event={"ID":"39d7d334-d9ba-4857-9161-3c5fc602194b","Type":"ContainerStarted","Data":"0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842"} Oct 30 21:54:46.192037 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:46.186005 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" event={"ID":"39d7d334-d9ba-4857-9161-3c5fc602194b","Type":"ContainerStarted","Data":"09f05e0d2c776ed4ce27fb006ab062bd43cd29640f349e611011b605fa2ece2a"} Oct 30 21:54:46.192037 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:46.186063 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 21:54:46.303495 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:46.303448 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" podStartSLOduration=2.303434925 podStartE2EDuration="2.303434925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:46.300432222 +0000 UTC m=+743.673058967" watchObservedRunningTime="2026-10-30 21:54:46.303434925 +0000 UTC m=+743.676061670" Oct 30 21:54:46.740539 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:46.740507 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9708a9f-23d1-4164-8d2c-78b74c4fd25c" path="/var/lib/kubelet/pods/a9708a9f-23d1-4164-8d2c-78b74c4fd25c/volumes" Oct 30 21:54:49.920838 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:49.920807 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:54:50.186637 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:50.186563 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t"] Oct 30 21:54:50.235803 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:50.235772 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj"] Oct 30 21:54:55.186870 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.186816 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" podUID="66d3da08-3bef-4edb-8dd8-fa181eb448ca" containerName="payload-processing" containerID="cri-o://4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2" gracePeriod=25 Oct 30 21:54:55.198679 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.198650 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh"] Oct 30 21:54:55.198887 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.198865 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" containerID="cri-o://d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b" gracePeriod=30 Oct 30 21:54:55.225317 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:54:55.225285 4859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d3da08_3bef_4edb_8dd8_fa181eb448ca.slice/crio-4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:54:55.237066 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.237030 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" podUID="eed977cb-d19a-4c42-a2b6-187e57037a80" containerName="payload-pre-processing" containerID="cri-o://75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c" gracePeriod=25 Oct 30 21:54:55.488608 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.488584 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:54:55.516835 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.516811 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:54:55.568278 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.568256 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed977cb-d19a-4c42-a2b6-187e57037a80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed977cb-d19a-4c42-a2b6-187e57037a80-plugins-config-volume\") pod \"eed977cb-d19a-4c42-a2b6-187e57037a80\" (UID: \"eed977cb-d19a-4c42-a2b6-187e57037a80\") " Oct 30 21:54:55.568388 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.568291 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d3da08-3bef-4edb-8dd8-fa181eb448ca-kube-api-access-2jg6d\" (UniqueName: \"kubernetes.io/projected/66d3da08-3bef-4edb-8dd8-fa181eb448ca-kube-api-access-2jg6d\") pod \"66d3da08-3bef-4edb-8dd8-fa181eb448ca\" (UID: \"66d3da08-3bef-4edb-8dd8-fa181eb448ca\") " Oct 30 21:54:55.568388 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.568320 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed977cb-d19a-4c42-a2b6-187e57037a80-kube-api-access-tzjhh\" (UniqueName: \"kubernetes.io/projected/eed977cb-d19a-4c42-a2b6-187e57037a80-kube-api-access-tzjhh\") pod \"eed977cb-d19a-4c42-a2b6-187e57037a80\" (UID: \"eed977cb-d19a-4c42-a2b6-187e57037a80\") " Oct 30 21:54:55.568487 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.568409 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d3da08-3bef-4edb-8dd8-fa181eb448ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66d3da08-3bef-4edb-8dd8-fa181eb448ca-plugins-config-volume\") pod \"66d3da08-3bef-4edb-8dd8-fa181eb448ca\" (UID: \"66d3da08-3bef-4edb-8dd8-fa181eb448ca\") " Oct 30 21:54:55.568736 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.568707 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed977cb-d19a-4c42-a2b6-187e57037a80-plugins-config-volume" pod "eed977cb-d19a-4c42-a2b6-187e57037a80" (UID: "eed977cb-d19a-4c42-a2b6-187e57037a80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:55.568845 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.568734 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d3da08-3bef-4edb-8dd8-fa181eb448ca-plugins-config-volume" pod "66d3da08-3bef-4edb-8dd8-fa181eb448ca" (UID: "66d3da08-3bef-4edb-8dd8-fa181eb448ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:55.570360 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.570341 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d3da08-3bef-4edb-8dd8-fa181eb448ca-kube-api-access-2jg6d" pod "66d3da08-3bef-4edb-8dd8-fa181eb448ca" (UID: "66d3da08-3bef-4edb-8dd8-fa181eb448ca"). InnerVolumeSpecName "kube-api-access-2jg6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:55.570438 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.570382 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed977cb-d19a-4c42-a2b6-187e57037a80-kube-api-access-tzjhh" pod "eed977cb-d19a-4c42-a2b6-187e57037a80" (UID: "eed977cb-d19a-4c42-a2b6-187e57037a80"). InnerVolumeSpecName "kube-api-access-tzjhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:55.669749 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.669717 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed977cb-d19a-4c42-a2b6-187e57037a80-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:55.669749 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.669743 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jg6d\" (UniqueName: \"kubernetes.io/projected/66d3da08-3bef-4edb-8dd8-fa181eb448ca-kube-api-access-2jg6d\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:55.669749 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.669754 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzjhh\" (UniqueName: \"kubernetes.io/projected/eed977cb-d19a-4c42-a2b6-187e57037a80-kube-api-access-tzjhh\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:55.669989 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:55.669762 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66d3da08-3bef-4edb-8dd8-fa181eb448ca-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:56.172900 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.172866 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj"] Oct 30 21:54:56.173342 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.173328 4859 cpu_manager.go:414] "containerMap: removing container" podUID="66d3da08-3bef-4edb-8dd8-fa181eb448ca" containerName="payload-processing" Oct 30 21:54:56.173387 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.173343 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d3da08-3bef-4edb-8dd8-fa181eb448ca" containerName="payload-processing" Oct 30 21:54:56.173387 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.173359 4859 cpu_manager.go:414] "containerMap: removing container" podUID="eed977cb-d19a-4c42-a2b6-187e57037a80" containerName="payload-pre-processing" Oct 30 21:54:56.173387 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.173364 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed977cb-d19a-4c42-a2b6-187e57037a80" containerName="payload-pre-processing" Oct 30 21:54:56.173477 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.173448 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d3da08-3bef-4edb-8dd8-fa181eb448ca" containerName="payload-processing" Oct 30 21:54:56.173477 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.173456 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed977cb-d19a-4c42-a2b6-187e57037a80" containerName="payload-pre-processing" Oct 30 21:54:56.176833 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.176814 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.228997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.228976 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:56.229353 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.229128 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:56.229353 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.229222 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:56.229353 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.229289 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\"" Oct 30 21:54:56.232436 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.232412 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj"] Oct 30 21:54:56.250353 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.250332 4859 generic.go:356] "Generic (PLEG): container finished" podID="eed977cb-d19a-4c42-a2b6-187e57037a80" containerID="75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c" exitCode=0 Oct 30 21:54:56.250449 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.250368 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" Oct 30 21:54:56.250449 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.250410 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" event={"ID":"eed977cb-d19a-4c42-a2b6-187e57037a80","Type":"ContainerDied","Data":"75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c"} Oct 30 21:54:56.250545 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.250454 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj" event={"ID":"eed977cb-d19a-4c42-a2b6-187e57037a80","Type":"ContainerDied","Data":"1ffd3768d8db892e7c84c62f4ef94c6aeee942de91c97b999ffc2565778f3598"} Oct 30 21:54:56.250545 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.250483 4859 scope.go:122] "RemoveContainer" containerID="75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c" Oct 30 21:54:56.252165 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.252142 4859 generic.go:356] "Generic (PLEG): container finished" podID="66d3da08-3bef-4edb-8dd8-fa181eb448ca" containerID="4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2" exitCode=0 Oct 30 21:54:56.252263 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.252194 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" event={"ID":"66d3da08-3bef-4edb-8dd8-fa181eb448ca","Type":"ContainerDied","Data":"4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2"} Oct 30 21:54:56.252263 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.252229 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" event={"ID":"66d3da08-3bef-4edb-8dd8-fa181eb448ca","Type":"ContainerDied","Data":"d375e08763958449cd17b3000e74cbfed27af6c2060ae5b074867814661fe209"} Oct 30 21:54:56.252384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.252290 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t" Oct 30 21:54:56.267693 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.267671 4859 scope.go:122] "RemoveContainer" containerID="75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c" Oct 30 21:54:56.268882 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:54:56.268017 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c\": container with ID starting with 75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c not found: ID does not exist" containerID="75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c" Oct 30 21:54:56.268882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.268047 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c"} err="failed to get container status \"75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c\": rpc error: code = NotFound desc = could not find container \"75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c\": container with ID starting with 75c92312c3515df11c0b38b6cd4369bd079c0bf98f9bf4bd442870a46cef778c not found: ID does not exist" Oct 30 21:54:56.268882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.268064 4859 scope.go:122] "RemoveContainer" containerID="4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2" Oct 30 21:54:56.275304 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.275283 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.275410 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.275344 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.275502 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.275479 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7gj6\" (UniqueName: \"kubernetes.io/projected/32fb8b46-90de-4fd0-a809-a7e378951583-kube-api-access-s7gj6\") pod \"maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.278138 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.278102 4859 scope.go:122] "RemoveContainer" containerID="4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2" Oct 30 21:54:56.295180 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.295156 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl"] Oct 30 21:54:56.295482 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.295458 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" containerID="cri-o://11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4" gracePeriod=30 Oct 30 21:54:56.296110 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:54:56.295991 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2\": container with ID starting with 4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2 not found: ID does not exist" containerID="4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2" Oct 30 21:54:56.296216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.296190 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2"} err="failed to get container status \"4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2\": rpc error: code = NotFound desc = could not find container \"4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2\": container with ID starting with 4cfb2746194c375096bbe4896bf1ad7b47555658dbba3ddc5d415be3404f4ec2 not found: ID does not exist" Oct 30 21:54:56.376415 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.376390 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7gj6\" (UniqueName: \"kubernetes.io/projected/32fb8b46-90de-4fd0-a809-a7e378951583-kube-api-access-s7gj6\") pod \"maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.376528 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.376500 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.376591 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.376575 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.377390 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.377368 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.404968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.404311 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.416046 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.416012 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj"] Oct 30 21:54:56.416967 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.416933 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-282df361-b8756946c-4rvvj"] Oct 30 21:54:56.436757 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.436696 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7gj6\" (UniqueName: \"kubernetes.io/projected/32fb8b46-90de-4fd0-a809-a7e378951583-kube-api-access-s7gj6\") pod \"maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.490194 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.490173 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:56.518420 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.518395 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t"] Oct 30 21:54:56.520912 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.520891 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-282df361-6b5ffbdc95-8k96t"] Oct 30 21:54:56.739931 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.739860 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d3da08-3bef-4edb-8dd8-fa181eb448ca" path="/var/lib/kubelet/pods/66d3da08-3bef-4edb-8dd8-fa181eb448ca/volumes" Oct 30 21:54:56.740269 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.740255 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed977cb-d19a-4c42-a2b6-187e57037a80" path="/var/lib/kubelet/pods/eed977cb-d19a-4c42-a2b6-187e57037a80/volumes" Oct 30 21:54:56.897753 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:56.897725 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj"] Oct 30 21:54:56.898742 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:54:56.898718 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fb8b46_90de_4fd0_a809_a7e378951583.slice/crio-5897b1d2db905822b541ecf490f5c793bbd9cbdf31510bb2b141f54e611cd203 WatchSource:0}: Error finding container 5897b1d2db905822b541ecf490f5c793bbd9cbdf31510bb2b141f54e611cd203: Status 404 returned error can't find the container with id 5897b1d2db905822b541ecf490f5c793bbd9cbdf31510bb2b141f54e611cd203 Oct 30 21:54:57.195372 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:57.195351 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 21:54:57.261998 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:57.261968 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" event={"ID":"32fb8b46-90de-4fd0-a809-a7e378951583","Type":"ContainerStarted","Data":"ec9fe182740de58a116375e63a2c6ab8e987960fbdd56f47a83465d2a99a231e"} Oct 30 21:54:57.262323 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:57.262024 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" event={"ID":"32fb8b46-90de-4fd0-a809-a7e378951583","Type":"ContainerStarted","Data":"5897b1d2db905822b541ecf490f5c793bbd9cbdf31510bb2b141f54e611cd203"} Oct 30 21:54:57.424414 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:57.424354 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" podStartSLOduration=1.424335209 podStartE2EDuration="1.424335209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:57.421689853 +0000 UTC m=+754.794316599" watchObservedRunningTime="2026-10-30 21:54:57.424335209 +0000 UTC m=+754.796961955" Oct 30 21:54:58.270383 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.270348 4859 generic.go:356] "Generic (PLEG): container finished" podID="32fb8b46-90de-4fd0-a809-a7e378951583" containerID="ec9fe182740de58a116375e63a2c6ab8e987960fbdd56f47a83465d2a99a231e" exitCode=0 Oct 30 21:54:58.270751 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.270400 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" event={"ID":"32fb8b46-90de-4fd0-a809-a7e378951583","Type":"ContainerDied","Data":"ec9fe182740de58a116375e63a2c6ab8e987960fbdd56f47a83465d2a99a231e"} Oct 30 21:54:58.686969 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.686925 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g"] Oct 30 21:54:58.692304 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.692280 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.744209 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.744180 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-90e0d5-openshift-default-dockercfg-mfg6n\"" Oct 30 21:54:58.750715 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.750691 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g"] Oct 30 21:54:58.801179 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.801147 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zklll\" (UniqueName: \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-kube-api-access-zklll\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.801334 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.801190 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e69c90-2004-4bf2-94ac-a5030d369ac6-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.801451 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.801410 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.801569 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.801552 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.801686 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.801664 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.801921 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.801899 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.802025 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.801932 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.802025 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.801998 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.802108 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.802027 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.903282 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.903250 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.903437 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.903324 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zklll\" (UniqueName: \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-kube-api-access-zklll\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.903437 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.903353 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e69c90-2004-4bf2-94ac-a5030d369ac6-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.903511 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.903445 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.903829 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.903806 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.903869 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.903854 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.903904 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.903895 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.903958 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.903924 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.904103 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.904079 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e69c90-2004-4bf2-94ac-a5030d369ac6-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.904188 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.903683 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.904248 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.904188 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.904248 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.904220 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.904359 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.904335 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.904498 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.904478 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.906294 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.906265 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.906633 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.906612 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.960036 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.959985 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:58.960036 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:58.960000 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zklll\" (UniqueName: \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-kube-api-access-zklll\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:59.015148 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.014847 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:54:59.394994 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.394970 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:54:59.407272 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.407248 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:54:59.454271 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.454240 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g"] Oct 30 21:54:59.455704 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:54:59.455679 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e69c90_2004_4bf2_94ac_a5030d369ac6.slice/crio-a8297314b53d8c6247bfc1d12a4d518628832898b7cbe56fba32e2e6d55f016a WatchSource:0}: Error finding container a8297314b53d8c6247bfc1d12a4d518628832898b7cbe56fba32e2e6d55f016a: Status 404 returned error can't find the container with id a8297314b53d8c6247bfc1d12a4d518628832898b7cbe56fba32e2e6d55f016a Oct 30 21:54:59.458057 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.458024 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:59.458135 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.458089 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:59.458135 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.458122 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:59.511490 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.511457 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fb8b46-90de-4fd0-a809-a7e378951583-kube-api-access-s7gj6\" (UniqueName: \"kubernetes.io/projected/32fb8b46-90de-4fd0-a809-a7e378951583-kube-api-access-s7gj6\") pod \"32fb8b46-90de-4fd0-a809-a7e378951583\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " Oct 30 21:54:59.511643 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.511517 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-openshift-ca\") pod \"32fb8b46-90de-4fd0-a809-a7e378951583\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " Oct 30 21:54:59.511643 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.511561 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-xks-ca\") pod \"32fb8b46-90de-4fd0-a809-a7e378951583\" (UID: \"32fb8b46-90de-4fd0-a809-a7e378951583\") " Oct 30 21:54:59.513997 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.512777 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-openshift-ca" pod "32fb8b46-90de-4fd0-a809-a7e378951583" (UID: "32fb8b46-90de-4fd0-a809-a7e378951583"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:59.515247 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.515222 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fb8b46-90de-4fd0-a809-a7e378951583-kube-api-access-s7gj6" pod "32fb8b46-90de-4fd0-a809-a7e378951583" (UID: "32fb8b46-90de-4fd0-a809-a7e378951583"). InnerVolumeSpecName "kube-api-access-s7gj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:59.516289 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.516262 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-xks-ca" pod "32fb8b46-90de-4fd0-a809-a7e378951583" (UID: "32fb8b46-90de-4fd0-a809-a7e378951583"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:59.612928 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.612885 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-openshift-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:59.612928 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.612924 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fb8b46-90de-4fd0-a809-a7e378951583-maas-api-xks-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:59.612928 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:54:59.612954 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7gj6\" (UniqueName: \"kubernetes.io/projected/32fb8b46-90de-4fd0-a809-a7e378951583-kube-api-access-s7gj6\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:00.283350 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.283327 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" Oct 30 21:55:00.283481 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.283331 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj" event={"ID":"32fb8b46-90de-4fd0-a809-a7e378951583","Type":"ContainerDied","Data":"5897b1d2db905822b541ecf490f5c793bbd9cbdf31510bb2b141f54e611cd203"} Oct 30 21:55:00.283481 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.283461 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5897b1d2db905822b541ecf490f5c793bbd9cbdf31510bb2b141f54e611cd203" Oct 30 21:55:00.285017 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.284982 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" event={"ID":"06e69c90-2004-4bf2-94ac-a5030d369ac6","Type":"ContainerStarted","Data":"8e30beef849efec90473f50f0e8531fbb756aadb48d9d34fa55bbc4b0d980e74"} Oct 30 21:55:00.285102 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.285040 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" event={"ID":"06e69c90-2004-4bf2-94ac-a5030d369ac6","Type":"ContainerStarted","Data":"a8297314b53d8c6247bfc1d12a4d518628832898b7cbe56fba32e2e6d55f016a"} Oct 30 21:55:00.403741 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.403679 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" podStartSLOduration=2.403665023 podStartE2EDuration="2.403665023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:00.400370259 +0000 UTC m=+757.772997003" watchObservedRunningTime="2026-10-30 21:55:00.403665023 +0000 UTC m=+757.776291768" Oct 30 21:55:00.456013 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.455985 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:55:00.521807 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.521709 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-istio-token\" (UniqueName: \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-istio-token\") pod \"022d4791-881d-4f32-9531-f0ded16008f3\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " Oct 30 21:55:00.521807 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.521756 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/022d4791-881d-4f32-9531-f0ded16008f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022d4791-881d-4f32-9531-f0ded16008f3-istio-podinfo\") pod \"022d4791-881d-4f32-9531-f0ded16008f3\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " Oct 30 21:55:00.522057 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.521832 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-kube-api-access-8hvtv\" (UniqueName: \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-kube-api-access-8hvtv\") pod \"022d4791-881d-4f32-9531-f0ded16008f3\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " Oct 30 21:55:00.522057 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.521869 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-socket\") pod \"022d4791-881d-4f32-9531-f0ded16008f3\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " Oct 30 21:55:00.522057 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.521924 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/022d4791-881d-4f32-9531-f0ded16008f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022d4791-881d-4f32-9531-f0ded16008f3-istiod-ca-cert\") pod \"022d4791-881d-4f32-9531-f0ded16008f3\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " Oct 30 21:55:00.522057 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.521999 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-credential-socket\") pod \"022d4791-881d-4f32-9531-f0ded16008f3\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " Oct 30 21:55:00.522057 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.522046 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-certs\") pod \"022d4791-881d-4f32-9531-f0ded16008f3\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " Oct 30 21:55:00.522303 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.522077 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-data\") pod \"022d4791-881d-4f32-9531-f0ded16008f3\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " Oct 30 21:55:00.522303 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.522130 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-envoy\") pod \"022d4791-881d-4f32-9531-f0ded16008f3\" (UID: \"022d4791-881d-4f32-9531-f0ded16008f3\") " Oct 30 21:55:00.523045 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.522683 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-socket" pod "022d4791-881d-4f32-9531-f0ded16008f3" (UID: "022d4791-881d-4f32-9531-f0ded16008f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:00.523045 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.522757 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-credential-socket" pod "022d4791-881d-4f32-9531-f0ded16008f3" (UID: "022d4791-881d-4f32-9531-f0ded16008f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:00.523233 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.523073 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/022d4791-881d-4f32-9531-f0ded16008f3-istiod-ca-cert" pod "022d4791-881d-4f32-9531-f0ded16008f3" (UID: "022d4791-881d-4f32-9531-f0ded16008f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:00.523741 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.523699 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-certs" pod "022d4791-881d-4f32-9531-f0ded16008f3" (UID: "022d4791-881d-4f32-9531-f0ded16008f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:00.524112 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.524092 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-data" pod "022d4791-881d-4f32-9531-f0ded16008f3" (UID: "022d4791-881d-4f32-9531-f0ded16008f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:00.526402 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.526135 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-istio-token" pod "022d4791-881d-4f32-9531-f0ded16008f3" (UID: "022d4791-881d-4f32-9531-f0ded16008f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:00.526402 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.526337 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022d4791-881d-4f32-9531-f0ded16008f3-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:00.526402 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.526362 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:00.526402 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.526376 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:00.526402 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.526389 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:00.526402 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.526402 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:00.526665 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.526415 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:00.529782 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.529754 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-kube-api-access-8hvtv" pod "022d4791-881d-4f32-9531-f0ded16008f3" (UID: "022d4791-881d-4f32-9531-f0ded16008f3"). InnerVolumeSpecName "kube-api-access-8hvtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:00.534379 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.534356 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-envoy" pod "022d4791-881d-4f32-9531-f0ded16008f3" (UID: "022d4791-881d-4f32-9531-f0ded16008f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:00.534503 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.534371 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/022d4791-881d-4f32-9531-f0ded16008f3-istio-podinfo" pod "022d4791-881d-4f32-9531-f0ded16008f3" (UID: "022d4791-881d-4f32-9531-f0ded16008f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:00.627826 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.627793 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022d4791-881d-4f32-9531-f0ded16008f3-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:00.627826 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.627819 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022d4791-881d-4f32-9531-f0ded16008f3-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:00.627826 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:00.627829 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hvtv\" (UniqueName: \"kubernetes.io/projected/022d4791-881d-4f32-9531-f0ded16008f3-kube-api-access-8hvtv\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:01.015287 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.015250 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:55:01.020003 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.019975 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:55:01.292821 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.292745 4859 generic.go:356] "Generic (PLEG): container finished" podID="022d4791-881d-4f32-9531-f0ded16008f3" containerID="d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b" exitCode=0 Oct 30 21:55:01.292821 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.292791 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" Oct 30 21:55:01.293066 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.292818 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" event={"ID":"022d4791-881d-4f32-9531-f0ded16008f3","Type":"ContainerDied","Data":"d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b"} Oct 30 21:55:01.293066 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.292868 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh" event={"ID":"022d4791-881d-4f32-9531-f0ded16008f3","Type":"ContainerDied","Data":"fd2c9be6d8f74e9e87d18d4c8b55d482ceea035b00201a74c71f0254d8ee6155"} Oct 30 21:55:01.293066 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.292891 4859 scope.go:122] "RemoveContainer" containerID="d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b" Oct 30 21:55:01.293219 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.293206 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:55:01.294428 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.294407 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:55:01.302427 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.302409 4859 scope.go:122] "RemoveContainer" containerID="5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84" Oct 30 21:55:01.314574 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.314551 4859 scope.go:122] "RemoveContainer" containerID="d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b" Oct 30 21:55:01.314884 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:01.314866 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b\": container with ID starting with d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b not found: ID does not exist" containerID="d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b" Oct 30 21:55:01.314989 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.314894 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b"} err="failed to get container status \"d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b\": rpc error: code = NotFound desc = could not find container \"d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b\": container with ID starting with d53987bd7e53230aeed3763fc4d565cf9f77afea28f0c322e4258107d24aff7b not found: ID does not exist" Oct 30 21:55:01.314989 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.314912 4859 scope.go:122] "RemoveContainer" containerID="5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84" Oct 30 21:55:01.315378 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:01.315196 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84\": container with ID starting with 5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84 not found: ID does not exist" containerID="5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84" Oct 30 21:55:01.315378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.315227 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84"} err="failed to get container status \"5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84\": rpc error: code = NotFound desc = could not find container \"5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84\": container with ID starting with 5bf1e583c2f85939ec60bbd5a8b44f9de70a7183c30f9763fac475999751df84 not found: ID does not exist" Oct 30 21:55:01.457969 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.457915 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh"] Oct 30 21:55:01.461412 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.461383 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f20e6e5e-openshift-default-7f98b64d97-g75fh"] Oct 30 21:55:01.570227 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.570206 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:55:01.637413 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637378 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-certs\") pod \"44663585-04f9-473f-bba5-37c8fbe63094\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " Oct 30 21:55:01.637413 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637412 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44663585-04f9-473f-bba5-37c8fbe63094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44663585-04f9-473f-bba5-37c8fbe63094-istio-podinfo\") pod \"44663585-04f9-473f-bba5-37c8fbe63094\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " Oct 30 21:55:01.637624 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637447 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-envoy\") pod \"44663585-04f9-473f-bba5-37c8fbe63094\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " Oct 30 21:55:01.637624 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637518 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-credential-socket\") pod \"44663585-04f9-473f-bba5-37c8fbe63094\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " Oct 30 21:55:01.637624 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637568 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-istio-token\" (UniqueName: \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-istio-token\") pod \"44663585-04f9-473f-bba5-37c8fbe63094\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " Oct 30 21:55:01.637624 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637616 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-socket\") pod \"44663585-04f9-473f-bba5-37c8fbe63094\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " Oct 30 21:55:01.637830 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637652 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-kube-api-access-xgtjp\" (UniqueName: \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-kube-api-access-xgtjp\") pod \"44663585-04f9-473f-bba5-37c8fbe63094\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " Oct 30 21:55:01.637830 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637702 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-data\") pod \"44663585-04f9-473f-bba5-37c8fbe63094\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " Oct 30 21:55:01.637830 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637757 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44663585-04f9-473f-bba5-37c8fbe63094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44663585-04f9-473f-bba5-37c8fbe63094-istiod-ca-cert\") pod \"44663585-04f9-473f-bba5-37c8fbe63094\" (UID: \"44663585-04f9-473f-bba5-37c8fbe63094\") " Oct 30 21:55:01.638043 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637696 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-certs" pod "44663585-04f9-473f-bba5-37c8fbe63094" (UID: "44663585-04f9-473f-bba5-37c8fbe63094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:01.638043 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637856 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-socket" pod "44663585-04f9-473f-bba5-37c8fbe63094" (UID: "44663585-04f9-473f-bba5-37c8fbe63094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:01.638150 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.637994 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-credential-socket" pod "44663585-04f9-473f-bba5-37c8fbe63094" (UID: "44663585-04f9-473f-bba5-37c8fbe63094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:01.638150 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.638046 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-data" pod "44663585-04f9-473f-bba5-37c8fbe63094" (UID: "44663585-04f9-473f-bba5-37c8fbe63094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:01.638252 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.638227 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:01.638308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.638249 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:01.638308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.638259 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44663585-04f9-473f-bba5-37c8fbe63094-istiod-ca-cert" pod "44663585-04f9-473f-bba5-37c8fbe63094" (UID: "44663585-04f9-473f-bba5-37c8fbe63094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:01.638308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.638264 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:01.638308 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.638291 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:01.640004 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.639974 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44663585-04f9-473f-bba5-37c8fbe63094-istio-podinfo" pod "44663585-04f9-473f-bba5-37c8fbe63094" (UID: "44663585-04f9-473f-bba5-37c8fbe63094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:01.640507 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.640484 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-envoy" pod "44663585-04f9-473f-bba5-37c8fbe63094" (UID: "44663585-04f9-473f-bba5-37c8fbe63094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:01.640624 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.640570 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-istio-token" pod "44663585-04f9-473f-bba5-37c8fbe63094" (UID: "44663585-04f9-473f-bba5-37c8fbe63094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:01.640624 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.640587 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-kube-api-access-xgtjp" pod "44663585-04f9-473f-bba5-37c8fbe63094" (UID: "44663585-04f9-473f-bba5-37c8fbe63094"). InnerVolumeSpecName "kube-api-access-xgtjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:01.739378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.739344 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgtjp\" (UniqueName: \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-kube-api-access-xgtjp\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:01.739378 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.739377 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44663585-04f9-473f-bba5-37c8fbe63094-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:01.739552 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.739387 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44663585-04f9-473f-bba5-37c8fbe63094-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:01.739552 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.739396 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44663585-04f9-473f-bba5-37c8fbe63094-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:01.739552 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:01.739405 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44663585-04f9-473f-bba5-37c8fbe63094-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:02.300432 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.300402 4859 generic.go:356] "Generic (PLEG): container finished" podID="44663585-04f9-473f-bba5-37c8fbe63094" containerID="11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4" exitCode=0 Oct 30 21:55:02.300573 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.300446 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" Oct 30 21:55:02.300573 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.300487 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" event={"ID":"44663585-04f9-473f-bba5-37c8fbe63094","Type":"ContainerDied","Data":"11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4"} Oct 30 21:55:02.300573 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.300534 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl" event={"ID":"44663585-04f9-473f-bba5-37c8fbe63094","Type":"ContainerDied","Data":"d7e153ab5531484305380e392de3ecb7e5be4961ec202c71934744cf2a16b121"} Oct 30 21:55:02.300573 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.300547 4859 scope.go:122] "RemoveContainer" containerID="11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4" Oct 30 21:55:02.310626 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.310607 4859 scope.go:122] "RemoveContainer" containerID="486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf" Oct 30 21:55:02.318855 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.318832 4859 scope.go:122] "RemoveContainer" containerID="11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4" Oct 30 21:55:02.319128 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:02.319105 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4\": container with ID starting with 11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4 not found: ID does not exist" containerID="11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4" Oct 30 21:55:02.319197 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.319135 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4"} err="failed to get container status \"11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4\": rpc error: code = NotFound desc = could not find container \"11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4\": container with ID starting with 11782a6a6ca9e0e21f868722934f86fe4e9a8ca2446cfcb43f652366207130d4 not found: ID does not exist" Oct 30 21:55:02.319197 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.319150 4859 scope.go:122] "RemoveContainer" containerID="486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf" Oct 30 21:55:02.319395 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:02.319379 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf\": container with ID starting with 486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf not found: ID does not exist" containerID="486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf" Oct 30 21:55:02.319435 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.319402 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf"} err="failed to get container status \"486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf\": rpc error: code = NotFound desc = could not find container \"486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf\": container with ID starting with 486f5b745e9ad5b12eeb3a0273663c5e92ce93c300cff62225a6f734a220e5cf not found: ID does not exist" Oct 30 21:55:02.464615 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.464592 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl"] Oct 30 21:55:02.469002 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.468984 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-282df361-openshift-default-66dc8d7489-ntrhl"] Oct 30 21:55:02.747799 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.741609 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="022d4791-881d-4f32-9531-f0ded16008f3" path="/var/lib/kubelet/pods/022d4791-881d-4f32-9531-f0ded16008f3/volumes" Oct 30 21:55:02.747799 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:02.742357 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44663585-04f9-473f-bba5-37c8fbe63094" path="/var/lib/kubelet/pods/44663585-04f9-473f-bba5-37c8fbe63094/volumes" Oct 30 21:55:06.929304 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:06.929226 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p"] Oct 30 21:55:07.036415 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:07.036383 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd"] Oct 30 21:55:09.875612 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:09.875579 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:55:11.296455 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.296418 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq"] Oct 30 21:55:11.297118 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297095 4859 cpu_manager.go:414] "containerMap: removing container" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" Oct 30 21:55:11.297214 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297119 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" Oct 30 21:55:11.297214 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297133 4859 cpu_manager.go:414] "containerMap: removing container" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" Oct 30 21:55:11.297214 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297140 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" Oct 30 21:55:11.297214 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297161 4859 cpu_manager.go:414] "containerMap: removing container" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" Oct 30 21:55:11.297214 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297168 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" Oct 30 21:55:11.297214 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297181 4859 cpu_manager.go:414] "containerMap: removing container" podUID="32fb8b46-90de-4fd0-a809-a7e378951583" containerName="revoke-keys" Oct 30 21:55:11.297214 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297187 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fb8b46-90de-4fd0-a809-a7e378951583" containerName="revoke-keys" Oct 30 21:55:11.297539 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297311 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" Oct 30 21:55:11.297539 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297326 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fb8b46-90de-4fd0-a809-a7e378951583" containerName="revoke-keys" Oct 30 21:55:11.297539 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297340 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" Oct 30 21:55:11.297539 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297355 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" Oct 30 21:55:11.297539 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.297363 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="44663585-04f9-473f-bba5-37c8fbe63094" containerName="istio-proxy" Oct 30 21:55:11.300759 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.300735 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 21:55:11.360131 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.360106 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq"] Oct 30 21:55:11.404914 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.404855 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-90e0d5-dockercfg-b44n8\"" Oct 30 21:55:11.404914 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.404883 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-90e0d5\"" Oct 30 21:55:11.442819 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.442771 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwdl\" (UniqueName: \"kubernetes.io/projected/2f093449-0fc4-40b1-a47b-2dfdcd448e07-kube-api-access-vzwdl\") pod \"payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq\" (UID: \"2f093449-0fc4-40b1-a47b-2dfdcd448e07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 21:55:11.443085 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.443057 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f093449-0fc4-40b1-a47b-2dfdcd448e07-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq\" (UID: \"2f093449-0fc4-40b1-a47b-2dfdcd448e07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 21:55:11.457711 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.457678 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn"] Oct 30 21:55:11.458275 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.458247 4859 cpu_manager.go:414] "containerMap: removing container" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" Oct 30 21:55:11.458275 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.458267 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022d4791-881d-4f32-9531-f0ded16008f3" containerName="istio-proxy" Oct 30 21:55:11.461641 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.461620 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 21:55:11.514667 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.514642 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn"] Oct 30 21:55:11.544260 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.544233 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f838b548-a378-4880-a793-bcdefc06f68b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn\" (UID: \"f838b548-a378-4880-a793-bcdefc06f68b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 21:55:11.544365 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.544281 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f093449-0fc4-40b1-a47b-2dfdcd448e07-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq\" (UID: \"2f093449-0fc4-40b1-a47b-2dfdcd448e07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 21:55:11.544415 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.544373 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6fr\" (UniqueName: \"kubernetes.io/projected/f838b548-a378-4880-a793-bcdefc06f68b-kube-api-access-xc6fr\") pod \"payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn\" (UID: \"f838b548-a378-4880-a793-bcdefc06f68b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 21:55:11.544450 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.544411 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwdl\" (UniqueName: \"kubernetes.io/projected/2f093449-0fc4-40b1-a47b-2dfdcd448e07-kube-api-access-vzwdl\") pod \"payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq\" (UID: \"2f093449-0fc4-40b1-a47b-2dfdcd448e07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 21:55:11.544923 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.544907 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f093449-0fc4-40b1-a47b-2dfdcd448e07-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq\" (UID: \"2f093449-0fc4-40b1-a47b-2dfdcd448e07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 21:55:11.600597 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.600569 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwdl\" (UniqueName: \"kubernetes.io/projected/2f093449-0fc4-40b1-a47b-2dfdcd448e07-kube-api-access-vzwdl\") pod \"payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq\" (UID: \"2f093449-0fc4-40b1-a47b-2dfdcd448e07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 21:55:11.614435 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.614413 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 21:55:11.644966 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.644923 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6fr\" (UniqueName: \"kubernetes.io/projected/f838b548-a378-4880-a793-bcdefc06f68b-kube-api-access-xc6fr\") pod \"payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn\" (UID: \"f838b548-a378-4880-a793-bcdefc06f68b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 21:55:11.645078 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.645041 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f838b548-a378-4880-a793-bcdefc06f68b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn\" (UID: \"f838b548-a378-4880-a793-bcdefc06f68b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 21:55:11.645625 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.645606 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f838b548-a378-4880-a793-bcdefc06f68b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn\" (UID: \"f838b548-a378-4880-a793-bcdefc06f68b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 21:55:11.702694 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.702658 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6fr\" (UniqueName: \"kubernetes.io/projected/f838b548-a378-4880-a793-bcdefc06f68b-kube-api-access-xc6fr\") pod \"payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn\" (UID: \"f838b548-a378-4880-a793-bcdefc06f68b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 21:55:11.775007 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.774978 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 21:55:11.930457 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.930369 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" podUID="9417396d-56c7-45a6-b419-1664028587ca" containerName="payload-processing" containerID="cri-o://9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957" gracePeriod=25 Oct 30 21:55:11.996513 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:11.996488 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq"] Oct 30 21:55:11.996993 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:55:11.996964 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f093449_0fc4_40b1_a47b_2dfdcd448e07.slice/crio-0cc07f67cc2e76af857b8ac9b4f88cb52459b8dd074fe92b773f9c55b4897f2d WatchSource:0}: Error finding container 0cc07f67cc2e76af857b8ac9b4f88cb52459b8dd074fe92b773f9c55b4897f2d: Status 404 returned error can't find the container with id 0cc07f67cc2e76af857b8ac9b4f88cb52459b8dd074fe92b773f9c55b4897f2d Oct 30 21:55:12.036645 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.036612 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" podUID="5a604d62-838f-4207-9879-ab3688c30412" containerName="payload-pre-processing" containerID="cri-o://b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d" gracePeriod=25 Oct 30 21:55:12.150143 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.150112 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn"] Oct 30 21:55:12.155288 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:55:12.154745 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf838b548_a378_4880_a793_bcdefc06f68b.slice/crio-87af8fdadc2361027ea9e734bf2f2098d55e560bbd3afcdabbf830df36c2520b WatchSource:0}: Error finding container 87af8fdadc2361027ea9e734bf2f2098d55e560bbd3afcdabbf830df36c2520b: Status 404 returned error can't find the container with id 87af8fdadc2361027ea9e734bf2f2098d55e560bbd3afcdabbf830df36c2520b Oct 30 21:55:12.164666 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.164645 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:55:12.251412 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.251384 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9417396d-56c7-45a6-b419-1664028587ca-kube-api-access-prr7f\" (UniqueName: \"kubernetes.io/projected/9417396d-56c7-45a6-b419-1664028587ca-kube-api-access-prr7f\") pod \"9417396d-56c7-45a6-b419-1664028587ca\" (UID: \"9417396d-56c7-45a6-b419-1664028587ca\") " Oct 30 21:55:12.251562 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.251453 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9417396d-56c7-45a6-b419-1664028587ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9417396d-56c7-45a6-b419-1664028587ca-plugins-config-volume\") pod \"9417396d-56c7-45a6-b419-1664028587ca\" (UID: \"9417396d-56c7-45a6-b419-1664028587ca\") " Oct 30 21:55:12.252051 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.252024 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9417396d-56c7-45a6-b419-1664028587ca-plugins-config-volume" pod "9417396d-56c7-45a6-b419-1664028587ca" (UID: "9417396d-56c7-45a6-b419-1664028587ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:12.254908 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.254882 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9417396d-56c7-45a6-b419-1664028587ca-kube-api-access-prr7f" pod "9417396d-56c7-45a6-b419-1664028587ca" (UID: "9417396d-56c7-45a6-b419-1664028587ca"). InnerVolumeSpecName "kube-api-access-prr7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:12.298259 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.298234 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:55:12.352381 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.352354 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a604d62-838f-4207-9879-ab3688c30412-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a604d62-838f-4207-9879-ab3688c30412-plugins-config-volume\") pod \"5a604d62-838f-4207-9879-ab3688c30412\" (UID: \"5a604d62-838f-4207-9879-ab3688c30412\") " Oct 30 21:55:12.352504 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.352458 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a604d62-838f-4207-9879-ab3688c30412-kube-api-access-8mgr6\" (UniqueName: \"kubernetes.io/projected/5a604d62-838f-4207-9879-ab3688c30412-kube-api-access-8mgr6\") pod \"5a604d62-838f-4207-9879-ab3688c30412\" (UID: \"5a604d62-838f-4207-9879-ab3688c30412\") " Oct 30 21:55:12.352692 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.352673 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prr7f\" (UniqueName: \"kubernetes.io/projected/9417396d-56c7-45a6-b419-1664028587ca-kube-api-access-prr7f\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:12.352692 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.352690 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9417396d-56c7-45a6-b419-1664028587ca-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:12.352853 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.352685 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a604d62-838f-4207-9879-ab3688c30412-plugins-config-volume" pod "5a604d62-838f-4207-9879-ab3688c30412" (UID: "5a604d62-838f-4207-9879-ab3688c30412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:12.354398 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.354376 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a604d62-838f-4207-9879-ab3688c30412-kube-api-access-8mgr6" pod "5a604d62-838f-4207-9879-ab3688c30412" (UID: "5a604d62-838f-4207-9879-ab3688c30412"). InnerVolumeSpecName "kube-api-access-8mgr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:12.368274 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.368244 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" event={"ID":"f838b548-a378-4880-a793-bcdefc06f68b","Type":"ContainerStarted","Data":"8a166d3ff2c7af068e2bf7b2ec21c508cd82a0e29a7c9e3530f7e61cf79b7820"} Oct 30 21:55:12.368370 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.368307 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" event={"ID":"f838b548-a378-4880-a793-bcdefc06f68b","Type":"ContainerStarted","Data":"87af8fdadc2361027ea9e734bf2f2098d55e560bbd3afcdabbf830df36c2520b"} Oct 30 21:55:12.368370 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.368343 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 21:55:12.369501 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.369480 4859 generic.go:356] "Generic (PLEG): container finished" podID="9417396d-56c7-45a6-b419-1664028587ca" containerID="9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957" exitCode=0 Oct 30 21:55:12.369566 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.369513 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" Oct 30 21:55:12.369614 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.369558 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" event={"ID":"9417396d-56c7-45a6-b419-1664028587ca","Type":"ContainerDied","Data":"9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957"} Oct 30 21:55:12.369614 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.369599 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p" event={"ID":"9417396d-56c7-45a6-b419-1664028587ca","Type":"ContainerDied","Data":"50165e7937d570a4d9945059db6a1d9337e79162328fae7bf5e76842fe9a0138"} Oct 30 21:55:12.369704 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.369623 4859 scope.go:122] "RemoveContainer" containerID="9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957" Oct 30 21:55:12.371129 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.371094 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" event={"ID":"2f093449-0fc4-40b1-a47b-2dfdcd448e07","Type":"ContainerStarted","Data":"8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85"} Oct 30 21:55:12.371218 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.371153 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 21:55:12.371218 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.371172 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" event={"ID":"2f093449-0fc4-40b1-a47b-2dfdcd448e07","Type":"ContainerStarted","Data":"0cc07f67cc2e76af857b8ac9b4f88cb52459b8dd074fe92b773f9c55b4897f2d"} Oct 30 21:55:12.381238 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.381207 4859 generic.go:356] "Generic (PLEG): container finished" podID="5a604d62-838f-4207-9879-ab3688c30412" containerID="b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d" exitCode=0 Oct 30 21:55:12.381320 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.381261 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" Oct 30 21:55:12.381320 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.381283 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" event={"ID":"5a604d62-838f-4207-9879-ab3688c30412","Type":"ContainerDied","Data":"b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d"} Oct 30 21:55:12.381386 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.381360 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd" event={"ID":"5a604d62-838f-4207-9879-ab3688c30412","Type":"ContainerDied","Data":"70b6fd33f65c75206023db0d2a84fe16c902ffcaeb42fd2a4d20a8fed8dbfabb"} Oct 30 21:55:12.385666 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.385642 4859 scope.go:122] "RemoveContainer" containerID="9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957" Oct 30 21:55:12.385963 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:12.385922 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957\": container with ID starting with 9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957 not found: ID does not exist" containerID="9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957" Oct 30 21:55:12.386018 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.385973 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957"} err="failed to get container status \"9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957\": rpc error: code = NotFound desc = could not find container \"9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957\": container with ID starting with 9899287ce2aab6711bd01fc5bcf119aaabc6e510b7a29fbe28364e678f490957 not found: ID does not exist" Oct 30 21:55:12.386018 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.385996 4859 scope.go:122] "RemoveContainer" containerID="b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d" Oct 30 21:55:12.394555 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.394534 4859 scope.go:122] "RemoveContainer" containerID="b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d" Oct 30 21:55:12.394803 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:12.394788 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d\": container with ID starting with b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d not found: ID does not exist" containerID="b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d" Oct 30 21:55:12.394845 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.394809 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d"} err="failed to get container status \"b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d\": rpc error: code = NotFound desc = could not find container \"b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d\": container with ID starting with b3eda54b8356382d0b1c6859f45902d3b313606c34f1024049814e38ef8ac05d not found: ID does not exist" Oct 30 21:55:12.453873 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.453801 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mgr6\" (UniqueName: \"kubernetes.io/projected/5a604d62-838f-4207-9879-ab3688c30412-kube-api-access-8mgr6\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:12.453873 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.453827 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a604d62-838f-4207-9879-ab3688c30412-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:12.529238 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.529195 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" podStartSLOduration=1.529183585 podStartE2EDuration="1.529183585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:12.527210841 +0000 UTC m=+769.899837581" watchObservedRunningTime="2026-10-30 21:55:12.529183585 +0000 UTC m=+769.901810328" Oct 30 21:55:12.638260 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.638235 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p"] Oct 30 21:55:12.644077 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.644052 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c678f1ee-696947bfcc-5r85p"] Oct 30 21:55:12.739875 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.739802 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9417396d-56c7-45a6-b419-1664028587ca" path="/var/lib/kubelet/pods/9417396d-56c7-45a6-b419-1664028587ca/volumes" Oct 30 21:55:12.754446 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.754402 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" podStartSLOduration=1.754389362 podStartE2EDuration="1.754389362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:12.752902502 +0000 UTC m=+770.125529249" watchObservedRunningTime="2026-10-30 21:55:12.754389362 +0000 UTC m=+770.127016125" Oct 30 21:55:12.864203 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.864173 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd"] Oct 30 21:55:12.867702 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:12.867674 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c678f1ee-6f4f646dc-w8xrd"] Oct 30 21:55:14.740741 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:14.740707 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a604d62-838f-4207-9879-ab3688c30412" path="/var/lib/kubelet/pods/5a604d62-838f-4207-9879-ab3688c30412/volumes" Oct 30 21:55:15.099220 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:15.099184 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:55:18.685577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.685544 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc"] Oct 30 21:55:18.686039 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.686025 4859 cpu_manager.go:414] "containerMap: removing container" podUID="5a604d62-838f-4207-9879-ab3688c30412" containerName="payload-pre-processing" Oct 30 21:55:18.686096 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.686040 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a604d62-838f-4207-9879-ab3688c30412" containerName="payload-pre-processing" Oct 30 21:55:18.686096 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.686061 4859 cpu_manager.go:414] "containerMap: removing container" podUID="9417396d-56c7-45a6-b419-1664028587ca" containerName="payload-processing" Oct 30 21:55:18.686096 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.686066 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9417396d-56c7-45a6-b419-1664028587ca" containerName="payload-processing" Oct 30 21:55:18.686184 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.686149 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a604d62-838f-4207-9879-ab3688c30412" containerName="payload-pre-processing" Oct 30 21:55:18.686184 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.686159 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9417396d-56c7-45a6-b419-1664028587ca" containerName="payload-processing" Oct 30 21:55:18.690869 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.690851 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 21:55:18.748520 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.748489 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc"] Oct 30 21:55:18.796361 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.796329 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d2a3e0\"" Oct 30 21:55:18.796773 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.796755 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-d2a3e0-dockercfg-5gnwj\"" Oct 30 21:55:18.824096 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.824057 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc\" (UID: \"59fc40d6-745b-42b8-9918-cb6ae52ff6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 21:55:18.824299 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.824190 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbsf\" (UniqueName: \"kubernetes.io/projected/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-kube-api-access-wjbsf\") pod \"payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc\" (UID: \"59fc40d6-745b-42b8-9918-cb6ae52ff6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 21:55:18.854566 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.854534 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n"] Oct 30 21:55:18.858366 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.858345 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 21:55:18.918879 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.918850 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n"] Oct 30 21:55:18.925531 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.925504 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc\" (UID: \"59fc40d6-745b-42b8-9918-cb6ae52ff6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 21:55:18.925675 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.925562 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbsf\" (UniqueName: \"kubernetes.io/projected/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-kube-api-access-wjbsf\") pod \"payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc\" (UID: \"59fc40d6-745b-42b8-9918-cb6ae52ff6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 21:55:18.926467 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.926441 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc\" (UID: \"59fc40d6-745b-42b8-9918-cb6ae52ff6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 21:55:18.982577 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:18.982508 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbsf\" (UniqueName: \"kubernetes.io/projected/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-kube-api-access-wjbsf\") pod \"payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc\" (UID: \"59fc40d6-745b-42b8-9918-cb6ae52ff6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 21:55:19.004145 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.004109 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 21:55:19.026768 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.026741 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hq6\" (UniqueName: \"kubernetes.io/projected/3db3fd91-9283-4244-966d-4ac6378ce512-kube-api-access-m2hq6\") pod \"payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n\" (UID: \"3db3fd91-9283-4244-966d-4ac6378ce512\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 21:55:19.026888 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.026792 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db3fd91-9283-4244-966d-4ac6378ce512-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n\" (UID: \"3db3fd91-9283-4244-966d-4ac6378ce512\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 21:55:19.128536 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.128492 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hq6\" (UniqueName: \"kubernetes.io/projected/3db3fd91-9283-4244-966d-4ac6378ce512-kube-api-access-m2hq6\") pod \"payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n\" (UID: \"3db3fd91-9283-4244-966d-4ac6378ce512\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 21:55:19.128726 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.128561 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db3fd91-9283-4244-966d-4ac6378ce512-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n\" (UID: \"3db3fd91-9283-4244-966d-4ac6378ce512\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 21:55:19.129310 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.129290 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db3fd91-9283-4244-966d-4ac6378ce512-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n\" (UID: \"3db3fd91-9283-4244-966d-4ac6378ce512\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 21:55:19.185857 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.185824 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hq6\" (UniqueName: \"kubernetes.io/projected/3db3fd91-9283-4244-966d-4ac6378ce512-kube-api-access-m2hq6\") pod \"payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n\" (UID: \"3db3fd91-9283-4244-966d-4ac6378ce512\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 21:55:19.376777 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.376751 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc"] Oct 30 21:55:19.378449 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:55:19.378421 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fc40d6_745b_42b8_9918_cb6ae52ff6aa.slice/crio-ffd92b0cac3a789427c8ca2ff9b53704fabd31edee3783a0a5fa1a7b4203758e WatchSource:0}: Error finding container ffd92b0cac3a789427c8ca2ff9b53704fabd31edee3783a0a5fa1a7b4203758e: Status 404 returned error can't find the container with id ffd92b0cac3a789427c8ca2ff9b53704fabd31edee3783a0a5fa1a7b4203758e Oct 30 21:55:19.427542 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.427511 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" event={"ID":"59fc40d6-745b-42b8-9918-cb6ae52ff6aa","Type":"ContainerStarted","Data":"ffd92b0cac3a789427c8ca2ff9b53704fabd31edee3783a0a5fa1a7b4203758e"} Oct 30 21:55:19.482316 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.482287 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 21:55:19.857885 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:19.857851 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n"] Oct 30 21:55:19.859389 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:55:19.859359 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db3fd91_9283_4244_966d_4ac6378ce512.slice/crio-109dc6553ac902c3b2ddcb3f3f87fb5694e13d124c7547f4544c06fd324613ff WatchSource:0}: Error finding container 109dc6553ac902c3b2ddcb3f3f87fb5694e13d124c7547f4544c06fd324613ff: Status 404 returned error can't find the container with id 109dc6553ac902c3b2ddcb3f3f87fb5694e13d124c7547f4544c06fd324613ff Oct 30 21:55:20.435780 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:20.435745 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" event={"ID":"59fc40d6-745b-42b8-9918-cb6ae52ff6aa","Type":"ContainerStarted","Data":"57b7e07abf9e1c5f7a9aec72c7fc81b772bc6b12ae00686541aa44fc4ea29b42"} Oct 30 21:55:20.436008 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:20.435812 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 21:55:20.437330 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:20.437302 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" event={"ID":"3db3fd91-9283-4244-966d-4ac6378ce512","Type":"ContainerStarted","Data":"1774386f8bfcb0ab71fd7a30de901c566c201cb4626a4e3bd548b0ad338ecb8c"} Oct 30 21:55:20.437431 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:20.437350 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" event={"ID":"3db3fd91-9283-4244-966d-4ac6378ce512","Type":"ContainerStarted","Data":"109dc6553ac902c3b2ddcb3f3f87fb5694e13d124c7547f4544c06fd324613ff"} Oct 30 21:55:20.437470 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:20.437430 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 21:55:20.549982 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:20.549915 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" podStartSLOduration=2.549899048 podStartE2EDuration="2.549899048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:20.549711616 +0000 UTC m=+777.922338358" watchObservedRunningTime="2026-10-30 21:55:20.549899048 +0000 UTC m=+777.922525795" Oct 30 21:55:20.674338 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:20.666797 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" podStartSLOduration=2.666776454 podStartE2EDuration="2.666776454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:20.663491453 +0000 UTC m=+778.036118197" watchObservedRunningTime="2026-10-30 21:55:20.666776454 +0000 UTC m=+778.039403201" Oct 30 21:55:23.392441 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:23.392410 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 21:55:23.392792 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:23.392487 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 21:55:24.397874 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.397838 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp"] Oct 30 21:55:24.403326 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.403309 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.456363 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.456337 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:24.456525 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.456337 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:24.506165 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.506133 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-90e0d5-serving-cert\"" Oct 30 21:55:24.506472 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.506453 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:55:24.506544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.506533 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-489tq\"" Oct 30 21:55:24.509148 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.509128 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp"] Oct 30 21:55:24.584731 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.584704 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aed1e80-3040-4b04-b64c-00c3c1db2da6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.584894 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.584746 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-metrics-certs\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.584894 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.584766 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx2pj\" (UniqueName: \"kubernetes.io/projected/3aed1e80-3040-4b04-b64c-00c3c1db2da6-kube-api-access-qx2pj\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.585054 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.584894 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.685708 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.685625 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aed1e80-3040-4b04-b64c-00c3c1db2da6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.685708 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.685677 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-metrics-certs\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.685708 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.685699 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx2pj\" (UniqueName: \"kubernetes.io/projected/3aed1e80-3040-4b04-b64c-00c3c1db2da6-kube-api-access-qx2pj\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.685927 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.685732 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.685927 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:24.685855 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-serving-cert: secret "maas-api-e2e-worker-w2-90e0d5-serving-cert" not found Oct 30 21:55:24.685927 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:24.685923 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-maas-api-tls podName:3aed1e80-3040-4b04-b64c-00c3c1db2da6 nodeName:}" failed. No retries permitted until 2026-10-30 21:55:25.185902441 +0000 UTC m=+782.558529179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-maas-api-tls") pod "maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" (UID: "3aed1e80-3040-4b04-b64c-00c3c1db2da6") : secret "maas-api-e2e-worker-w2-90e0d5-serving-cert" not found Oct 30 21:55:24.686358 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.686330 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aed1e80-3040-4b04-b64c-00c3c1db2da6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.688157 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.688133 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-metrics-certs\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:24.748511 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:24.743836 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx2pj\" (UniqueName: \"kubernetes.io/projected/3aed1e80-3040-4b04-b64c-00c3c1db2da6-kube-api-access-qx2pj\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:25.190710 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:25.190668 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:25.193128 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:25.193095 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:25.315775 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:25.315743 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:25.708610 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:25.708582 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp"] Oct 30 21:55:25.709957 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:55:25.709919 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aed1e80_3040_4b04_b64c_00c3c1db2da6.slice/crio-dc3bb69073dc3886ac9a7067734eefd58304b9cf43710d7bdd9236120fbe5f1b WatchSource:0}: Error finding container dc3bb69073dc3886ac9a7067734eefd58304b9cf43710d7bdd9236120fbe5f1b: Status 404 returned error can't find the container with id dc3bb69073dc3886ac9a7067734eefd58304b9cf43710d7bdd9236120fbe5f1b Oct 30 21:55:26.474555 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:26.474513 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" event={"ID":"3aed1e80-3040-4b04-b64c-00c3c1db2da6","Type":"ContainerStarted","Data":"dc3bb69073dc3886ac9a7067734eefd58304b9cf43710d7bdd9236120fbe5f1b"} Oct 30 21:55:29.497187 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:29.497147 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" event={"ID":"3aed1e80-3040-4b04-b64c-00c3c1db2da6","Type":"ContainerStarted","Data":"d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081"} Oct 30 21:55:29.497585 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:29.497211 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:29.612858 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:29.612692 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" podStartSLOduration=2.56182859 podStartE2EDuration="5.612668087s" totalImagesPullingTime="3.050839497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:29.61110431 +0000 UTC m=+786.983731057" watchObservedRunningTime="2026-10-30 21:55:29.612668087 +0000 UTC m=+786.985294833" Oct 30 21:55:31.446634 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:31.446604 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 21:55:31.447139 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:31.446684 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 21:55:31.693738 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:31.693701 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj"] Oct 30 21:55:31.704629 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:31.704535 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c678f1ee-mq9tj"] Oct 30 21:55:31.812290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:31.812253 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g"] Oct 30 21:55:31.812531 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:31.812508 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" containerID="cri-o://f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0" gracePeriod=30 Oct 30 21:55:32.741356 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:32.741324 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32fb8b46-90de-4fd0-a809-a7e378951583" path="/var/lib/kubelet/pods/32fb8b46-90de-4fd0-a809-a7e378951583/volumes" Oct 30 21:55:33.147644 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.147602 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k"] Oct 30 21:55:33.154515 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.154483 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.179732 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.179698 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.179963 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.179740 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.179963 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.179865 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.180084 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.179961 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.180084 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.180008 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.180084 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.180069 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.180241 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.180107 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.180241 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.180155 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.180347 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.180277 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8mz\" (UniqueName: \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-kube-api-access-lb8mz\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.217883 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.217850 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k"] Oct 30 21:55:33.253416 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.253375 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1c3eef-openshift-default-dockercfg-hgpgr\"" Oct 30 21:55:33.280905 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.280869 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281119 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.280932 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281119 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.280984 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281119 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281003 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281119 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281023 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281119 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281064 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281377 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281126 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8mz\" (UniqueName: \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-kube-api-access-lb8mz\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281377 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281182 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281377 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281213 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281532 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281397 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281532 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281463 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281532 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281519 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281742 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281721 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.281909 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.281885 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.283348 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.283325 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.283750 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.283734 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.316507 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.316471 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:55:33.346235 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.346203 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.369391 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.369349 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8mz\" (UniqueName: \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-kube-api-access-lb8mz\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.473026 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.472707 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:33.943711 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:55:33.943677 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f5d232_4abf_4adc_9a62_2aa9be7b9fb7.slice/crio-741df6166069c03b8111284f8db629f3581dc98ae2083b1327d010ed5d2a79fa WatchSource:0}: Error finding container 741df6166069c03b8111284f8db629f3581dc98ae2083b1327d010ed5d2a79fa: Status 404 returned error can't find the container with id 741df6166069c03b8111284f8db629f3581dc98ae2083b1327d010ed5d2a79fa Oct 30 21:55:33.945151 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.944725 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k"] Oct 30 21:55:33.951205 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.951155 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:33.951361 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.951247 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:33.951361 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:33.951292 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:34.538110 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:34.538075 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" event={"ID":"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7","Type":"ContainerStarted","Data":"22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c"} Oct 30 21:55:34.538381 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:34.538362 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" event={"ID":"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7","Type":"ContainerStarted","Data":"741df6166069c03b8111284f8db629f3581dc98ae2083b1327d010ed5d2a79fa"} Oct 30 21:55:34.679968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:34.676648 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" podStartSLOduration=1.676628799 podStartE2EDuration="1.676628799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:34.67444884 +0000 UTC m=+792.047075591" watchObservedRunningTime="2026-10-30 21:55:34.676628799 +0000 UTC m=+792.049255546" Oct 30 21:55:35.484968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:35.484316 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:35.491388 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:35.491358 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:35.508431 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:35.508401 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 21:55:35.544230 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:35.544195 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:35.545348 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:35.545325 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:55:37.119042 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.119019 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:55:37.155207 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155175 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-data\") pod \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " Oct 30 21:55:37.155207 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155211 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istiod-ca-cert\") pod \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " Oct 30 21:55:37.155390 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155271 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-credential-socket\") pod \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " Oct 30 21:55:37.155390 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155309 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-podinfo\") pod \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " Oct 30 21:55:37.155390 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155337 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-kube-api-access-nnx8k\" (UniqueName: \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-kube-api-access-nnx8k\") pod \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " Oct 30 21:55:37.155390 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155376 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-certs\") pod \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " Oct 30 21:55:37.155538 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155471 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-data" pod "43b4dfab-3cc0-4287-a018-dbf68d67f9b0" (UID: "43b4dfab-3cc0-4287-a018-dbf68d67f9b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:37.155592 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155556 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-credential-socket" pod "43b4dfab-3cc0-4287-a018-dbf68d67f9b0" (UID: "43b4dfab-3cc0-4287-a018-dbf68d67f9b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:37.155592 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155581 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-envoy\") pod \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " Oct 30 21:55:37.155692 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155616 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-socket\") pod \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " Oct 30 21:55:37.155692 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155657 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-token\" (UniqueName: \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-token\") pod \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\" (UID: \"43b4dfab-3cc0-4287-a018-dbf68d67f9b0\") " Oct 30 21:55:37.155794 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155683 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istiod-ca-cert" pod "43b4dfab-3cc0-4287-a018-dbf68d67f9b0" (UID: "43b4dfab-3cc0-4287-a018-dbf68d67f9b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:37.155794 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155734 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-certs" pod "43b4dfab-3cc0-4287-a018-dbf68d67f9b0" (UID: "43b4dfab-3cc0-4287-a018-dbf68d67f9b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:37.155922 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.155895 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-socket" pod "43b4dfab-3cc0-4287-a018-dbf68d67f9b0" (UID: "43b4dfab-3cc0-4287-a018-dbf68d67f9b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:37.156111 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.156089 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:37.156218 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.156115 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:37.156218 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.156130 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:37.156218 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.156144 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:37.156218 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.156157 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:37.157509 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.157488 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-podinfo" pod "43b4dfab-3cc0-4287-a018-dbf68d67f9b0" (UID: "43b4dfab-3cc0-4287-a018-dbf68d67f9b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:37.157596 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.157573 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-envoy" pod "43b4dfab-3cc0-4287-a018-dbf68d67f9b0" (UID: "43b4dfab-3cc0-4287-a018-dbf68d67f9b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:37.157996 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.157976 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-token" pod "43b4dfab-3cc0-4287-a018-dbf68d67f9b0" (UID: "43b4dfab-3cc0-4287-a018-dbf68d67f9b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:37.158296 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.158277 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-kube-api-access-nnx8k" pod "43b4dfab-3cc0-4287-a018-dbf68d67f9b0" (UID: "43b4dfab-3cc0-4287-a018-dbf68d67f9b0"). InnerVolumeSpecName "kube-api-access-nnx8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:37.257368 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.257334 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:37.257368 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.257362 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:37.257368 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.257371 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:37.257600 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.257381 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnx8k\" (UniqueName: \"kubernetes.io/projected/43b4dfab-3cc0-4287-a018-dbf68d67f9b0-kube-api-access-nnx8k\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:37.559391 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.559305 4859 generic.go:356] "Generic (PLEG): container finished" podID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerID="f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0" exitCode=0 Oct 30 21:55:37.559391 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.559359 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" Oct 30 21:55:37.559579 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.559385 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" event={"ID":"43b4dfab-3cc0-4287-a018-dbf68d67f9b0","Type":"ContainerDied","Data":"f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0"} Oct 30 21:55:37.559579 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.559444 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g" event={"ID":"43b4dfab-3cc0-4287-a018-dbf68d67f9b0","Type":"ContainerDied","Data":"c79ba669667f433c3b912e1507fe477eed4542b2f08f42b60a11b99ee3de24bf"} Oct 30 21:55:37.559579 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.559474 4859 scope.go:122] "RemoveContainer" containerID="f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0" Oct 30 21:55:37.596226 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.596206 4859 scope.go:122] "RemoveContainer" containerID="13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84" Oct 30 21:55:37.606628 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.606606 4859 scope.go:122] "RemoveContainer" containerID="f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0" Oct 30 21:55:37.606968 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:37.606926 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0\": container with ID starting with f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0 not found: ID does not exist" containerID="f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0" Oct 30 21:55:37.607082 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.606971 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0"} err="failed to get container status \"f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0\": rpc error: code = NotFound desc = could not find container \"f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0\": container with ID starting with f66c01f9eff9ed544f7316379165498e43e881574013f086ad90b2eac8fd0ab0 not found: ID does not exist" Oct 30 21:55:37.607082 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.606990 4859 scope.go:122] "RemoveContainer" containerID="13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84" Oct 30 21:55:37.607297 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:37.607276 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84\": container with ID starting with 13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84 not found: ID does not exist" containerID="13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84" Oct 30 21:55:37.607351 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.607308 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84"} err="failed to get container status \"13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84\": rpc error: code = NotFound desc = could not find container \"13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84\": container with ID starting with 13292cbb2ed430c340f22fccab1a167e68d8ec0368bf2dca0bc56c70780b4b84 not found: ID does not exist" Oct 30 21:55:37.738475 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.738440 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g"] Oct 30 21:55:37.741739 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:37.741717 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c678f1ee-openshift-default-67f6bdd7c-gvm8g"] Oct 30 21:55:38.740741 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:38.740710 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" path="/var/lib/kubelet/pods/43b4dfab-3cc0-4287-a018-dbf68d67f9b0/volumes" Oct 30 21:55:39.876376 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:55:39.876338 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:55:41.152242 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.152205 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn"] Oct 30 21:55:41.152864 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.152846 4859 cpu_manager.go:414] "containerMap: removing container" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" Oct 30 21:55:41.152911 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.152865 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" Oct 30 21:55:41.152911 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.152882 4859 cpu_manager.go:414] "containerMap: removing container" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" Oct 30 21:55:41.152911 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.152887 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" Oct 30 21:55:41.153020 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.152986 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" Oct 30 21:55:41.153020 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.152997 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b4dfab-3cc0-4287-a018-dbf68d67f9b0" containerName="istio-proxy" Oct 30 21:55:41.156384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.156367 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 21:55:41.209232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.209203 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-1c3eef\"" Oct 30 21:55:41.227489 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.227435 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998baa49-be51-4cd6-95d3-b7e6288f974e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn\" (UID: \"998baa49-be51-4cd6-95d3-b7e6288f974e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 21:55:41.227489 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.227493 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbxkg\" (UniqueName: \"kubernetes.io/projected/998baa49-be51-4cd6-95d3-b7e6288f974e-kube-api-access-zbxkg\") pod \"payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn\" (UID: \"998baa49-be51-4cd6-95d3-b7e6288f974e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 21:55:41.261755 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.261717 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn"] Oct 30 21:55:41.308614 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.308580 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw"] Oct 30 21:55:41.313401 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.313368 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 21:55:41.329017 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.328988 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998baa49-be51-4cd6-95d3-b7e6288f974e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn\" (UID: \"998baa49-be51-4cd6-95d3-b7e6288f974e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 21:55:41.329184 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.329026 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbxkg\" (UniqueName: \"kubernetes.io/projected/998baa49-be51-4cd6-95d3-b7e6288f974e-kube-api-access-zbxkg\") pod \"payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn\" (UID: \"998baa49-be51-4cd6-95d3-b7e6288f974e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 21:55:41.329626 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.329607 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998baa49-be51-4cd6-95d3-b7e6288f974e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn\" (UID: \"998baa49-be51-4cd6-95d3-b7e6288f974e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 21:55:41.351061 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.351006 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw"] Oct 30 21:55:41.366152 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.366121 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-1c3eef-dockercfg-mxt5n\"" Oct 30 21:55:41.388889 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.388857 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbxkg\" (UniqueName: \"kubernetes.io/projected/998baa49-be51-4cd6-95d3-b7e6288f974e-kube-api-access-zbxkg\") pod \"payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn\" (UID: \"998baa49-be51-4cd6-95d3-b7e6288f974e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 21:55:41.430211 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.430129 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw\" (UID: \"1f3d68b6-bfaa-4994-9ace-6eedae9df25f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 21:55:41.430211 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.430167 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6m9d\" (UniqueName: \"kubernetes.io/projected/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-kube-api-access-m6m9d\") pod \"payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw\" (UID: \"1f3d68b6-bfaa-4994-9ace-6eedae9df25f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 21:55:41.472990 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.472926 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 21:55:41.530893 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.530850 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw\" (UID: \"1f3d68b6-bfaa-4994-9ace-6eedae9df25f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 21:55:41.531026 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.530908 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6m9d\" (UniqueName: \"kubernetes.io/projected/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-kube-api-access-m6m9d\") pod \"payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw\" (UID: \"1f3d68b6-bfaa-4994-9ace-6eedae9df25f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 21:55:41.532091 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.532032 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw\" (UID: \"1f3d68b6-bfaa-4994-9ace-6eedae9df25f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 21:55:41.588739 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.588699 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6m9d\" (UniqueName: \"kubernetes.io/projected/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-kube-api-access-m6m9d\") pod \"payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw\" (UID: \"1f3d68b6-bfaa-4994-9ace-6eedae9df25f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 21:55:41.628663 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.628622 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 21:55:41.858477 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:41.858451 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn"] Oct 30 21:55:41.859462 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:55:41.859433 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998baa49_be51_4cd6_95d3_b7e6288f974e.slice/crio-cb1c654dc48bcb5dc0f8174cbbf54e4d6c060af16bc31528636434f5e9b8bdc0 WatchSource:0}: Error finding container cb1c654dc48bcb5dc0f8174cbbf54e4d6c060af16bc31528636434f5e9b8bdc0: Status 404 returned error can't find the container with id cb1c654dc48bcb5dc0f8174cbbf54e4d6c060af16bc31528636434f5e9b8bdc0 Oct 30 21:55:42.007638 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:42.007609 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw"] Oct 30 21:55:42.010213 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:55:42.010184 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3d68b6_bfaa_4994_9ace_6eedae9df25f.slice/crio-6d277185bab64a75e637b0e9c17b4ee0aeabe54cd4e13579337f90883fa02380 WatchSource:0}: Error finding container 6d277185bab64a75e637b0e9c17b4ee0aeabe54cd4e13579337f90883fa02380: Status 404 returned error can't find the container with id 6d277185bab64a75e637b0e9c17b4ee0aeabe54cd4e13579337f90883fa02380 Oct 30 21:55:42.621306 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:42.621262 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" event={"ID":"1f3d68b6-bfaa-4994-9ace-6eedae9df25f","Type":"ContainerStarted","Data":"4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa"} Oct 30 21:55:42.622703 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:42.621355 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" event={"ID":"1f3d68b6-bfaa-4994-9ace-6eedae9df25f","Type":"ContainerStarted","Data":"6d277185bab64a75e637b0e9c17b4ee0aeabe54cd4e13579337f90883fa02380"} Oct 30 21:55:42.622703 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:42.621474 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 21:55:42.624914 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:42.623404 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" event={"ID":"998baa49-be51-4cd6-95d3-b7e6288f974e","Type":"ContainerStarted","Data":"7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd"} Oct 30 21:55:42.624914 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:42.623467 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" event={"ID":"998baa49-be51-4cd6-95d3-b7e6288f974e","Type":"ContainerStarted","Data":"cb1c654dc48bcb5dc0f8174cbbf54e4d6c060af16bc31528636434f5e9b8bdc0"} Oct 30 21:55:42.624914 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:42.623597 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 21:55:42.738753 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:42.738517 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" podStartSLOduration=1.73849972 podStartE2EDuration="1.73849972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:42.737800369 +0000 UTC m=+800.110427113" watchObservedRunningTime="2026-10-30 21:55:42.73849972 +0000 UTC m=+800.111126466" Oct 30 21:55:42.859349 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:42.859283 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" podStartSLOduration=1.859261177 podStartE2EDuration="1.859261177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:42.852635394 +0000 UTC m=+800.225262154" watchObservedRunningTime="2026-10-30 21:55:42.859261177 +0000 UTC m=+800.231887924" Oct 30 21:55:43.573782 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:43.573734 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:55:43.631568 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:43.631536 4859 generic.go:356] "Generic (PLEG): container finished" podID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerID="8e30beef849efec90473f50f0e8531fbb756aadb48d9d34fa55bbc4b0d980e74" exitCode=0 Oct 30 21:55:43.632095 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:43.631603 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" event={"ID":"06e69c90-2004-4bf2-94ac-a5030d369ac6","Type":"ContainerDied","Data":"8e30beef849efec90473f50f0e8531fbb756aadb48d9d34fa55bbc4b0d980e74"} Oct 30 21:55:43.632095 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:43.632027 4859 scope.go:122] "RemoveContainer" containerID="8e30beef849efec90473f50f0e8531fbb756aadb48d9d34fa55bbc4b0d980e74" Oct 30 21:55:43.633279 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:43.633248 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:43.633371 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:43.633317 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:43.633371 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:43.633349 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:44.639496 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:44.639455 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" event={"ID":"06e69c90-2004-4bf2-94ac-a5030d369ac6","Type":"ContainerStarted","Data":"928a09032ef8dac95cf573e67859bfad56e0377885d8a1ec41b4be70c5a5ba3b"} Oct 30 21:55:45.015685 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:45.015598 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:55:45.016989 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:45.016958 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:55:45.017115 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:45.017006 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:55:46.015309 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:46.015272 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:55:46.015703 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:46.015324 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:55:46.293125 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:46.293032 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:55:47.015937 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:47.015901 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:55:47.016432 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:47.015975 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:55:48.016223 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:48.016188 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:55:48.016614 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:48.016240 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:55:49.015960 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:49.015911 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:55:49.016139 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:49.015991 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:55:50.019494 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:50.019460 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:55:50.020603 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:50.020581 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 21:55:53.635560 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:53.635530 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 21:55:53.636062 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:55:53.635674 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 21:56:03.194377 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.194334 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g"] Oct 30 21:56:03.201321 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.201293 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.260297 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.260264 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g"] Oct 30 21:56:03.271023 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.270983 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.271186 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.271045 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.271296 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.271192 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9x9\" (UniqueName: \"kubernetes.io/projected/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-kube-api-access-2q9x9\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.271296 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.271231 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-metrics-certs\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.305548 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.305509 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-1c3eef-serving-cert\"" Oct 30 21:56:03.372019 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.371973 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-metrics-certs\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.372271 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.372070 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.372271 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.372119 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.372271 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.372248 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9x9\" (UniqueName: \"kubernetes.io/projected/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-kube-api-access-2q9x9\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.372928 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:56:03.372574 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-serving-cert: secret "maas-api-e2e-worker-w1-1c3eef-serving-cert" not found Oct 30 21:56:03.372928 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:56:03.372646 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-maas-api-tls podName:4f8242e9-9b3b-429a-ad38-09dad5c6ef30 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:03.872626401 +0000 UTC m=+821.245253139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-maas-api-tls") pod "maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" (UID: "4f8242e9-9b3b-429a-ad38-09dad5c6ef30") : secret "maas-api-e2e-worker-w1-1c3eef-serving-cert" not found Oct 30 21:56:03.373754 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.373725 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.376149 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.376122 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-metrics-certs\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.430009 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.429977 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9x9\" (UniqueName: \"kubernetes.io/projected/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-kube-api-access-2q9x9\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.878036 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.877988 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:03.880611 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:03.880564 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:04.114262 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:04.114220 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:04.485734 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:04.485635 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:56:04.493103 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:56:04.493069 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8242e9_9b3b_429a_ad38_09dad5c6ef30.slice/crio-fa33576a534978db7e20415a1ac34abb54ca618cb548eb85f043c0cf7529924b WatchSource:0}: Error finding container fa33576a534978db7e20415a1ac34abb54ca618cb548eb85f043c0cf7529924b: Status 404 returned error can't find the container with id fa33576a534978db7e20415a1ac34abb54ca618cb548eb85f043c0cf7529924b Oct 30 21:56:04.536681 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:04.536636 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g"] Oct 30 21:56:04.808651 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:04.808550 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" event={"ID":"4f8242e9-9b3b-429a-ad38-09dad5c6ef30","Type":"ContainerStarted","Data":"fa33576a534978db7e20415a1ac34abb54ca618cb548eb85f043c0cf7529924b"} Oct 30 21:56:05.815921 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:05.815824 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" event={"ID":"4f8242e9-9b3b-429a-ad38-09dad5c6ef30","Type":"ContainerStarted","Data":"eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf"} Oct 30 21:56:05.816384 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:05.815985 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:05.934889 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:05.934838 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" podStartSLOduration=1.958402141 podStartE2EDuration="2.934823161s" totalImagesPullingTime="976.42102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:05.933334442 +0000 UTC m=+823.305961190" watchObservedRunningTime="2026-10-30 21:56:05.934823161 +0000 UTC m=+823.307449906" Oct 30 21:56:09.881025 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:56:09.880988 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:56:11.828374 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:11.828343 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 21:56:21.585128 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:21.585084 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:56:21.585589 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:21.585164 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:56:21.946015 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:21.945912 4859 generic.go:356] "Generic (PLEG): container finished" podID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerID="22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c" exitCode=0 Oct 30 21:56:21.946015 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:21.945981 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" event={"ID":"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7","Type":"ContainerDied","Data":"22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c"} Oct 30 21:56:21.946370 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:21.946355 4859 scope.go:122] "RemoveContainer" containerID="22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c" Oct 30 21:56:21.947601 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:21.947573 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:21.947649 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:21.947640 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:21.947682 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:21.947673 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:22.497549 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:22.497515 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:56:23.008965 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:23.002550 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" event={"ID":"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7","Type":"ContainerStarted","Data":"2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267"} Oct 30 21:56:23.487990 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:23.486241 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:56:23.487990 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:23.486365 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:56:23.487990 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:23.486398 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:56:24.476371 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:24.476333 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:56:24.476989 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:24.476913 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:56:25.476380 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:25.476339 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:56:25.477047 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:25.477014 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:56:26.479969 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:26.477036 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:56:26.479969 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:26.477113 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:56:27.477967 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:27.477319 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:56:27.477967 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:27.477382 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:56:28.477294 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:28.477262 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:56:28.477779 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:28.477647 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:56:28.478249 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:28.478230 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 21:56:39.876287 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:56:39.876215 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:56:46.148683 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:56:46.148645 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:57:09.878577 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:57:09.878537 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:57:15.313762 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:15.313725 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:57:22.866354 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:22.866321 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 21:57:22.868341 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:22.868314 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 21:57:22.873078 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:22.873055 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 21:57:22.875028 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:22.875009 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 21:57:36.363411 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:36.363376 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:57:39.876380 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:57:39.876345 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:57:40.517161 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:40.517128 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm"] Oct 30 21:57:40.517469 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:40.517418 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" containerID="cri-o://6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876" gracePeriod=30 Oct 30 21:57:40.566881 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:40.566845 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58"] Oct 30 21:57:40.567154 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:40.567122 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" podUID="b249f651-382f-4948-9fb1-455e46949f49" containerName="istio-proxy" containerID="cri-o://02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7" gracePeriod=30 Oct 30 21:57:42.710622 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:42.710584 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-xxpxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:42.711029 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:42.710648 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:45.835999 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.835973 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:57:45.842519 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.842493 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-kube-api-access-wql7j\" (UniqueName: \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-kube-api-access-wql7j\") pod \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " Oct 30 21:57:45.842659 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.842547 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-socket\") pod \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " Oct 30 21:57:45.842659 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.842583 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-credential-socket\") pod \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " Oct 30 21:57:45.842659 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.842619 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istiod-ca-cert\") pod \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " Oct 30 21:57:45.842659 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.842649 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-data\") pod \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " Oct 30 21:57:45.842882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.842675 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-certs\") pod \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " Oct 30 21:57:45.842882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.842717 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-podinfo\") pod \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " Oct 30 21:57:45.842882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.842777 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-token\" (UniqueName: \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-token\") pod \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " Oct 30 21:57:45.842882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.842819 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-envoy\") pod \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\" (UID: \"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103\") " Oct 30 21:57:45.843121 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.842878 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-socket" pod "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" (UID: "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:45.848408 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.843055 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istiod-ca-cert" pod "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" (UID: "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:45.848535 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.843107 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-certs" pod "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" (UID: "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:45.848535 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.843286 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-data" pod "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" (UID: "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:45.848535 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.843309 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.848535 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.845338 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-podinfo" pod "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" (UID: "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:45.848535 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.845539 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-kube-api-access-wql7j" pod "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" (UID: "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103"). InnerVolumeSpecName "kube-api-access-wql7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:45.848535 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.845756 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-token" pod "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" (UID: "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:45.848535 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.846041 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-envoy" pod "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" (UID: "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:45.848535 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.847102 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-credential-socket" pod "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" (UID: "8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:45.874908 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.874882 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:57:45.949667 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.949635 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-data\") pod \"b249f651-382f-4948-9fb1-455e46949f49\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " Oct 30 21:57:45.949848 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.949700 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-kube-api-access-7rm8s\" (UniqueName: \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-kube-api-access-7rm8s\") pod \"b249f651-382f-4948-9fb1-455e46949f49\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " Oct 30 21:57:45.949848 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.949739 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-envoy\") pod \"b249f651-382f-4948-9fb1-455e46949f49\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " Oct 30 21:57:45.949848 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.949765 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-socket\") pod \"b249f651-382f-4948-9fb1-455e46949f49\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " Oct 30 21:57:45.949848 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.949793 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-istio-token\" (UniqueName: \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-istio-token\") pod \"b249f651-382f-4948-9fb1-455e46949f49\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " Oct 30 21:57:45.950081 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.949818 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b249f651-382f-4948-9fb1-455e46949f49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b249f651-382f-4948-9fb1-455e46949f49-istio-podinfo\") pod \"b249f651-382f-4948-9fb1-455e46949f49\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " Oct 30 21:57:45.950081 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950034 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-credential-socket\") pod \"b249f651-382f-4948-9fb1-455e46949f49\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " Oct 30 21:57:45.950081 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950064 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-data" pod "b249f651-382f-4948-9fb1-455e46949f49" (UID: "b249f651-382f-4948-9fb1-455e46949f49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:45.950236 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950110 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-certs\") pod \"b249f651-382f-4948-9fb1-455e46949f49\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " Oct 30 21:57:45.950236 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950148 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b249f651-382f-4948-9fb1-455e46949f49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b249f651-382f-4948-9fb1-455e46949f49-istiod-ca-cert\") pod \"b249f651-382f-4948-9fb1-455e46949f49\" (UID: \"b249f651-382f-4948-9fb1-455e46949f49\") " Oct 30 21:57:45.950359 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950328 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-socket" pod "b249f651-382f-4948-9fb1-455e46949f49" (UID: "b249f651-382f-4948-9fb1-455e46949f49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:45.950525 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950510 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.950584 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950531 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.950584 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950546 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.950584 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950562 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wql7j\" (UniqueName: \"kubernetes.io/projected/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-kube-api-access-wql7j\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.950584 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950578 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.950780 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950586 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-credential-socket" pod "b249f651-382f-4948-9fb1-455e46949f49" (UID: "b249f651-382f-4948-9fb1-455e46949f49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:45.950780 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950592 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.950780 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950594 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b249f651-382f-4948-9fb1-455e46949f49-istiod-ca-cert" pod "b249f651-382f-4948-9fb1-455e46949f49" (UID: "b249f651-382f-4948-9fb1-455e46949f49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:45.950780 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950625 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.950780 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950642 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.950780 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950657 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.950780 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950672 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.950780 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.950752 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-certs" pod "b249f651-382f-4948-9fb1-455e46949f49" (UID: "b249f651-382f-4948-9fb1-455e46949f49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:45.952853 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.952826 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-istio-token" pod "b249f651-382f-4948-9fb1-455e46949f49" (UID: "b249f651-382f-4948-9fb1-455e46949f49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:45.953031 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.952898 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-kube-api-access-7rm8s" pod "b249f651-382f-4948-9fb1-455e46949f49" (UID: "b249f651-382f-4948-9fb1-455e46949f49"). InnerVolumeSpecName "kube-api-access-7rm8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:45.953111 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.953050 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-envoy" pod "b249f651-382f-4948-9fb1-455e46949f49" (UID: "b249f651-382f-4948-9fb1-455e46949f49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:45.953171 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:45.953126 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b249f651-382f-4948-9fb1-455e46949f49-istio-podinfo" pod "b249f651-382f-4948-9fb1-455e46949f49" (UID: "b249f651-382f-4948-9fb1-455e46949f49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:46.052237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.052151 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:46.052237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.052181 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:46.052237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.052195 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b249f651-382f-4948-9fb1-455e46949f49-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:46.052237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.052208 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rm8s\" (UniqueName: \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-kube-api-access-7rm8s\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:46.052237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.052221 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b249f651-382f-4948-9fb1-455e46949f49-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:46.052237 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.052233 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b249f651-382f-4948-9fb1-455e46949f49-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:46.052620 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.052245 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b249f651-382f-4948-9fb1-455e46949f49-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:46.596983 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.596034 4859 generic.go:356] "Generic (PLEG): container finished" podID="b249f651-382f-4948-9fb1-455e46949f49" containerID="02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7" exitCode=0 Oct 30 21:57:46.596983 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.596199 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" event={"ID":"b249f651-382f-4948-9fb1-455e46949f49","Type":"ContainerDied","Data":"02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7"} Oct 30 21:57:46.596983 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.596236 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" event={"ID":"b249f651-382f-4948-9fb1-455e46949f49","Type":"ContainerDied","Data":"6ae2e1c03b9d54903fb357cb639c5a54817f22ed4dc8b2d11127a32dd19245e3"} Oct 30 21:57:46.596983 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.596255 4859 scope.go:122] "RemoveContainer" containerID="02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7" Oct 30 21:57:46.596983 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.596366 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58" Oct 30 21:57:46.601395 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.601363 4859 generic.go:356] "Generic (PLEG): container finished" podID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerID="6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876" exitCode=0 Oct 30 21:57:46.601544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.601425 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" Oct 30 21:57:46.601544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.601443 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" event={"ID":"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103","Type":"ContainerDied","Data":"6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876"} Oct 30 21:57:46.601544 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.601492 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm" event={"ID":"8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103","Type":"ContainerDied","Data":"9fbb34fa58270baccd210a129efc3f8b820979260decc95ed054c05260e8c38f"} Oct 30 21:57:46.611163 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.611141 4859 scope.go:122] "RemoveContainer" containerID="02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7" Oct 30 21:57:46.611503 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:57:46.611481 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7\": container with ID starting with 02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7 not found: ID does not exist" containerID="02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7" Oct 30 21:57:46.611603 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.611518 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7"} err="failed to get container status \"02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7\": rpc error: code = NotFound desc = could not find container \"02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7\": container with ID starting with 02e2535e13f01282044529497e5db77c36ddf7be64c29080d3e7b32cdd878dd7 not found: ID does not exist" Oct 30 21:57:46.611603 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.611540 4859 scope.go:122] "RemoveContainer" containerID="6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876" Oct 30 21:57:46.621172 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.621151 4859 scope.go:122] "RemoveContainer" containerID="e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2" Oct 30 21:57:46.629405 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.629383 4859 scope.go:122] "RemoveContainer" containerID="6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876" Oct 30 21:57:46.629704 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:57:46.629682 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876\": container with ID starting with 6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876 not found: ID does not exist" containerID="6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876" Oct 30 21:57:46.629795 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.629724 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876"} err="failed to get container status \"6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876\": rpc error: code = NotFound desc = could not find container \"6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876\": container with ID starting with 6f0f09a4e7b6b6be366ee486deebe7db333013770f6ce44ca6f2076208853876 not found: ID does not exist" Oct 30 21:57:46.629795 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.629747 4859 scope.go:122] "RemoveContainer" containerID="e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2" Oct 30 21:57:46.630031 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:57:46.630008 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2\": container with ID starting with e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2 not found: ID does not exist" containerID="e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2" Oct 30 21:57:46.630077 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.630041 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2"} err="failed to get container status \"e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2\": rpc error: code = NotFound desc = could not find container \"e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2\": container with ID starting with e898356976e2851d3cd4c2066ad5c845d1ac96e355fce06ead1b4a0fbd3383d2 not found: ID does not exist" Oct 30 21:57:46.768315 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.768280 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58"] Oct 30 21:57:46.816069 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.815985 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dnh58"] Oct 30 21:57:46.895848 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.895812 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm"] Oct 30 21:57:46.904482 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:46.904452 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-xxpxm"] Oct 30 21:57:48.740179 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:48.740148 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" path="/var/lib/kubelet/pods/8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103/volumes" Oct 30 21:57:48.740587 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:48.740573 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b249f651-382f-4948-9fb1-455e46949f49" path="/var/lib/kubelet/pods/b249f651-382f-4948-9fb1-455e46949f49/volumes" Oct 30 21:57:49.085197 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.085116 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp"] Oct 30 21:57:49.085631 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.085617 4859 cpu_manager.go:414] "containerMap: removing container" podUID="b249f651-382f-4948-9fb1-455e46949f49" containerName="istio-proxy" Oct 30 21:57:49.085671 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.085632 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b249f651-382f-4948-9fb1-455e46949f49" containerName="istio-proxy" Oct 30 21:57:49.085671 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.085660 4859 cpu_manager.go:414] "containerMap: removing container" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" Oct 30 21:57:49.085671 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.085665 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" Oct 30 21:57:49.085758 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.085676 4859 cpu_manager.go:414] "containerMap: removing container" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" Oct 30 21:57:49.085758 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.085681 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" Oct 30 21:57:49.085758 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.085749 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b249f651-382f-4948-9fb1-455e46949f49" containerName="istio-proxy" Oct 30 21:57:49.085887 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.085761 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" Oct 30 21:57:49.090726 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.090701 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.154380 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.154352 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp"] Oct 30 21:57:49.180882 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.180855 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-tmp-dir\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.181048 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.180898 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-home\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.181048 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.180922 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvr2\" (UniqueName: \"kubernetes.io/projected/47691dfc-77d7-47dc-9922-d439e2de1fe2-kube-api-access-tdvr2\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.181164 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.181052 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47691dfc-77d7-47dc-9922-d439e2de1fe2-tls-certs\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.181164 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.181112 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-dshm\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.181272 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.181224 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-model-cache\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.199843 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.199821 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"kube-root-ca.crt\"" Oct 30 21:57:49.200101 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.200079 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"openshift-service-ca.crt\"" Oct 30 21:57:49.200228 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.200145 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"premium-w2-90e0d5-kserve-self-signed-certs\"" Oct 30 21:57:49.200228 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.200206 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"default-dockercfg-ft4lg\"" Oct 30 21:57:49.281868 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.281831 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-tmp-dir\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.282092 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.281877 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-home\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.282092 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.281903 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvr2\" (UniqueName: \"kubernetes.io/projected/47691dfc-77d7-47dc-9922-d439e2de1fe2-kube-api-access-tdvr2\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.282092 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.281933 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47691dfc-77d7-47dc-9922-d439e2de1fe2-tls-certs\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.282092 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.282001 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-dshm\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.282092 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.282050 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-model-cache\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.282380 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.282355 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-tmp-dir\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.282523 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.282499 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-home\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.282660 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.282637 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-model-cache\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.284637 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.284616 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47691dfc-77d7-47dc-9922-d439e2de1fe2-tls-certs\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.284735 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.284685 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-dshm\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.338787 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.338756 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvr2\" (UniqueName: \"kubernetes.io/projected/47691dfc-77d7-47dc-9922-d439e2de1fe2-kube-api-access-tdvr2\") pod \"premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.403783 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.403762 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:49.788407 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:49.788378 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp"] Oct 30 21:57:49.790474 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:57:49.790449 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47691dfc_77d7_47dc_9922_d439e2de1fe2.slice/crio-e0401e5e2bc96c8a14eded44099ddd73a58b77a7ce288652a6a69c09e8a495ee WatchSource:0}: Error finding container e0401e5e2bc96c8a14eded44099ddd73a58b77a7ce288652a6a69c09e8a495ee: Status 404 returned error can't find the container with id e0401e5e2bc96c8a14eded44099ddd73a58b77a7ce288652a6a69c09e8a495ee Oct 30 21:57:50.632492 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:50.632452 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" event={"ID":"47691dfc-77d7-47dc-9922-d439e2de1fe2","Type":"ContainerStarted","Data":"aa055bf1a7b368f8021c261d1c3ad6926c51afb6c9ade4eb9345a3f5f165dda3"} Oct 30 21:57:50.632681 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:50.632506 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" event={"ID":"47691dfc-77d7-47dc-9922-d439e2de1fe2","Type":"ContainerStarted","Data":"e0401e5e2bc96c8a14eded44099ddd73a58b77a7ce288652a6a69c09e8a495ee"} Oct 30 21:57:50.747883 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:50.747834 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podStartSLOduration=1.747820984 podStartE2EDuration="1.747820984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:50.745581551 +0000 UTC m=+928.118208309" watchObservedRunningTime="2026-10-30 21:57:50.747820984 +0000 UTC m=+928.120447730" Oct 30 21:57:51.404232 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:51.404199 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:57.858974 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:57.858921 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:57:59.289462 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.289430 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw"] Oct 30 21:57:59.290072 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.290057 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fcf5ae0-b25d-42d0-b0b3-1fedebf3a103" containerName="istio-proxy" Oct 30 21:57:59.293801 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.293780 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.345625 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.345589 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d2a3e0-openshift-default-dockercfg-t4vpk\"" Oct 30 21:57:59.388960 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.388916 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.389090 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.388985 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.389131 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.389109 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.389192 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.389175 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.389231 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.389213 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.389330 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.389312 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.389373 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.389359 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.389438 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.389419 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.389486 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.389471 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt4l6\" (UniqueName: \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-kube-api-access-tt4l6\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.397795 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.397762 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw"] Oct 30 21:57:59.404854 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.404834 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:59.417676 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.417654 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:57:59.490688 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.490658 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.490688 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.490696 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.490935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.490721 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt4l6\" (UniqueName: \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-kube-api-access-tt4l6\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.490935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.490780 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.490935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.490860 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.490935 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.490891 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.491192 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.490966 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.491192 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.491001 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.491192 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.491096 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.491192 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.491172 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.491411 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.491389 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.491464 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.491409 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.491789 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.491764 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.491862 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.491802 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.493507 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.493482 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.493597 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.493584 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.550420 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.550350 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt4l6\" (UniqueName: \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-kube-api-access-tt4l6\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.550420 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.550384 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.608108 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.608081 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:57:59.707324 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:57:59.707129 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 21:58:00.039743 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:00.039716 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw"] Oct 30 21:58:00.041586 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:58:00.041555 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77f54c3_1e05_4549_bdd9_d2a5a768d84f.slice/crio-9a3ea2e1e59774f7024752342dccb3166fb385185171deb831ec494c9aab012f WatchSource:0}: Error finding container 9a3ea2e1e59774f7024752342dccb3166fb385185171deb831ec494c9aab012f: Status 404 returned error can't find the container with id 9a3ea2e1e59774f7024752342dccb3166fb385185171deb831ec494c9aab012f Oct 30 21:58:00.044043 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:00.044002 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:00.044126 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:00.044086 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:00.044208 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:00.044127 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:00.700246 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:00.700214 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" event={"ID":"a77f54c3-1e05-4549-bdd9-d2a5a768d84f","Type":"ContainerStarted","Data":"1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527"} Oct 30 21:58:00.700635 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:00.700281 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" event={"ID":"a77f54c3-1e05-4549-bdd9-d2a5a768d84f","Type":"ContainerStarted","Data":"9a3ea2e1e59774f7024752342dccb3166fb385185171deb831ec494c9aab012f"} Oct 30 21:58:00.817610 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:00.817450 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" podStartSLOduration=1.817432228 podStartE2EDuration="1.817432228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:00.817265004 +0000 UTC m=+938.189891749" watchObservedRunningTime="2026-10-30 21:58:00.817432228 +0000 UTC m=+938.190058975" Oct 30 21:58:01.608531 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:01.608494 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:58:01.610324 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:01.610296 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:58:01.610447 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:01.610356 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:58:02.609385 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:02.609345 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:58:02.609781 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:02.609414 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:58:03.609439 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:03.609400 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:58:03.609921 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:03.609454 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:58:04.609337 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:04.609305 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:58:04.609499 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:04.609365 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:58:05.608472 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:05.608437 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:58:05.608644 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:05.608490 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:58:06.612782 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:06.612747 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:58:06.614620 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:06.614594 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:58:06.614925 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:06.614898 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 21:58:07.879510 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.879482 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772"] Oct 30 21:58:07.883621 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.883597 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:07.935400 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.935374 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-e90e35-openshift-default-dockercfg-4j6lz\"" Oct 30 21:58:07.983158 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.983113 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxg2l\" (UniqueName: \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-kube-api-access-gxg2l\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:07.983333 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.983216 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:07.983333 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.983284 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:07.983333 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.983324 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:07.983504 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.983352 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:07.983504 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.983378 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:07.983504 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.983427 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:07.983504 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.983500 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:07.983683 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.983533 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:07.989376 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:07.989344 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772"] Oct 30 21:58:08.084090 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084065 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084231 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084115 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxg2l\" (UniqueName: \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-kube-api-access-gxg2l\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084231 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084152 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084231 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084190 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084231 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084218 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084475 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084241 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084475 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084264 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084475 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084307 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084475 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084370 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084659 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084625 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084716 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084693 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.084912 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084890 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.085021 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.084886 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.085118 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.085095 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.086776 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.086757 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.086858 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.086811 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.140430 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.140383 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.140909 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.140886 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxg2l\" (UniqueName: \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-kube-api-access-gxg2l\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.197113 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.197078 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:08.621996 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.621970 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772"] Oct 30 21:58:08.623080 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:58:08.623057 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92deb8c0_dc69_4cba_af2a_a77ed60be8cc.slice/crio-314b3072f27fac1d574efcb12cd5f905aaba267e23866f701d1b3d338aa3b1a5 WatchSource:0}: Error finding container 314b3072f27fac1d574efcb12cd5f905aaba267e23866f701d1b3d338aa3b1a5: Status 404 returned error can't find the container with id 314b3072f27fac1d574efcb12cd5f905aaba267e23866f701d1b3d338aa3b1a5 Oct 30 21:58:08.625263 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.625227 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:08.625344 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.625294 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:08.625344 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.625320 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:08.762058 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.762024 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" event={"ID":"92deb8c0-dc69-4cba-af2a-a77ed60be8cc","Type":"ContainerStarted","Data":"27b804637390e175875e42aff7954a86578a67c7507399e8ff1e579fb407e195"} Oct 30 21:58:08.762206 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.762071 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" event={"ID":"92deb8c0-dc69-4cba-af2a-a77ed60be8cc","Type":"ContainerStarted","Data":"314b3072f27fac1d574efcb12cd5f905aaba267e23866f701d1b3d338aa3b1a5"} Oct 30 21:58:08.878672 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:08.878570 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" podStartSLOduration=1.878556879 podStartE2EDuration="1.878556879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:08.877363585 +0000 UTC m=+946.249990332" watchObservedRunningTime="2026-10-30 21:58:08.878556879 +0000 UTC m=+946.251183625" Oct 30 21:58:09.197505 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:09.197409 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:09.198892 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:09.198862 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:58:09.199041 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:09.198918 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:58:09.908802 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:58:09.908761 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:58:10.197589 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:10.197499 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:58:10.197589 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:10.197559 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:58:11.198301 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:11.198267 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:58:11.198768 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:11.198326 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:58:12.199364 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:12.199327 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:58:12.199961 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:12.199909 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:58:13.197596 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:13.197563 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:58:13.197768 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:13.197616 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:58:14.198526 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:14.198478 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:58:14.199049 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:14.198539 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:58:15.978629 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:15.978591 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:15.979143 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:15.978973 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:15.979711 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:15.979690 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 21:58:27.014057 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:58:27.014019 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:58:39.881165 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:58:39.881124 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:59:02.732057 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:02.732015 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:59:07.776133 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.776085 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5577b47559-4ll85"] Oct 30 21:59:07.781261 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.781238 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:07.790498 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.790469 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5577b47559-4ll85"] Oct 30 21:59:07.833566 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.833534 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 21:59:07.833799 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.833633 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 21:59:07.833799 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.833637 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 21:59:07.833799 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.833752 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 21:59:07.834088 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.833808 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-501bd36994fb\"" Oct 30 21:59:07.834088 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.833986 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7513640d12da\"" Oct 30 21:59:07.834198 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.834086 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5j2m4\"" Oct 30 21:59:07.899371 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.899339 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:07.899555 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.899378 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-plugins-config-volume\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:07.899555 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.899417 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t49q9\" (UniqueName: \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-kube-api-access-t49q9\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:07.899555 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:07.899540 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-routing-overlay-volume\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:08.000210 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:08.000171 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:08.000383 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:08.000224 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-plugins-config-volume\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:08.000383 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:08.000269 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t49q9\" (UniqueName: \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-kube-api-access-t49q9\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:08.000383 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:08.000304 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-routing-overlay-volume\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:08.001044 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:08.001020 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-routing-overlay-volume\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:08.001332 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:08.001305 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-plugins-config-volume\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:08.002983 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:08.002959 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:08.057123 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:08.057047 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t49q9\" (UniqueName: \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-kube-api-access-t49q9\") pod \"payload-processing-external-model-5577b47559-4ll85\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:08.098462 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:08.098430 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:08.485667 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:08.485632 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5577b47559-4ll85"] Oct 30 21:59:08.489676 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:59:08.489647 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e4f3dd_f33c_482b_add4_bde12575edd0.slice/crio-e9df374f475ae52e6217759e62a69b948d3c6ccdc3c78a249646b0dead302438 WatchSource:0}: Error finding container e9df374f475ae52e6217759e62a69b948d3c6ccdc3c78a249646b0dead302438: Status 404 returned error can't find the container with id e9df374f475ae52e6217759e62a69b948d3c6ccdc3c78a249646b0dead302438 Oct 30 21:59:09.277663 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:09.277624 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" event={"ID":"92e4f3dd-f33c-482b-add4-bde12575edd0","Type":"ContainerStarted","Data":"e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1"} Oct 30 21:59:09.278133 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:09.277689 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" event={"ID":"92e4f3dd-f33c-482b-add4-bde12575edd0","Type":"ContainerStarted","Data":"e9df374f475ae52e6217759e62a69b948d3c6ccdc3c78a249646b0dead302438"} Oct 30 21:59:09.278133 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:09.277739 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:09.418364 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:09.418294 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" podStartSLOduration=2.41827441 podStartE2EDuration="2.41827441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:09.416077618 +0000 UTC m=+1006.788704364" watchObservedRunningTime="2026-10-30 21:59:09.41827441 +0000 UTC m=+1006.790901157" Oct 30 21:59:09.880852 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:59:09.880812 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 21:59:20.261676 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.261641 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq"] Oct 30 21:59:20.266026 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.266001 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.287836 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.287808 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 21:59:20.318008 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.317972 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b8717e-openshift-default-dockercfg-pqzrg\"" Oct 30 21:59:20.373017 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.372986 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq"] Oct 30 21:59:20.439223 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.439192 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.439394 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.439250 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.439394 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.439326 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.439470 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.439400 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.439504 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.439484 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.439537 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.439515 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.439662 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.439649 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.439725 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.439698 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmv8t\" (UniqueName: \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-kube-api-access-mmv8t\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.439770 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.439741 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.540891 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.540797 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.540891 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.540847 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.540891 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.540867 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.540909 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.540937 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmv8t\" (UniqueName: \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-kube-api-access-mmv8t\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.540992 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.541010 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541181 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.541042 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541434 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.541202 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541434 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.541244 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541620 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.541599 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541677 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.541649 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541735 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.541649 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.541803 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.541785 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.543547 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.543510 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.543653 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.543547 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.599471 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.599437 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmv8t\" (UniqueName: \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-kube-api-access-mmv8t\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.600618 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.600588 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:20.880762 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:20.880724 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:21.324720 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:21.324474 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq"] Oct 30 21:59:21.325611 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:21.325562 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:21.325855 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:21.325830 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:21.326046 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:21.326026 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:21.352809 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:21.352771 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" event={"ID":"b67abee5-46f6-464d-9a37-8afaf3ebdd5b","Type":"ContainerStarted","Data":"9e4ed61c13701d52e59d1e45a7a94ef0a1995825f9fdf0321cc71c7b54ac7d75"} Oct 30 21:59:22.360889 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:22.360853 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" event={"ID":"b67abee5-46f6-464d-9a37-8afaf3ebdd5b","Type":"ContainerStarted","Data":"3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb"} Oct 30 21:59:22.599968 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:22.584379 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" podStartSLOduration=2.584360421 podStartE2EDuration="2.584360421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:22.531820484 +0000 UTC m=+1019.904447230" watchObservedRunningTime="2026-10-30 21:59:22.584360421 +0000 UTC m=+1019.956987169" Oct 30 21:59:22.881205 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:22.881116 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:22.883265 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:22.883234 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:59:22.883458 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:22.883433 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:59:23.882650 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:23.882614 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:59:23.883320 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:23.883274 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:59:24.883549 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:24.883504 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:59:24.884251 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:24.884205 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:59:25.511712 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.511674 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2"] Oct 30 21:59:25.515531 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.515510 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.566013 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.561033 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57"] Oct 30 21:59:25.586246 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.586218 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.610606 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.610578 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2"] Oct 30 21:59:25.628387 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.628356 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57"] Oct 30 21:59:25.637459 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.637432 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.637570 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.637492 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.637641 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.637619 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.637691 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.637671 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8242e4-ae99-448e-a660-a01fe5076393-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.637737 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.637707 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t94f\" (UniqueName: \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-kube-api-access-8t94f\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.637794 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.637774 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.637863 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.637847 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8242e4-ae99-448e-a660-a01fe5076393-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.637898 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.637875 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.637978 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.637938 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.738729 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.738692 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cce801-630f-4f59-8054-5bee59bde387-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.738900 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.738739 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.738900 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.738836 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.739018 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739001 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.739063 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739048 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739107 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739086 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cce801-630f-4f59-8054-5bee59bde387-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.739139 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739112 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739139 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739131 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.739202 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739178 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8242e4-ae99-448e-a660-a01fe5076393-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739238 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739206 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t94f\" (UniqueName: \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-kube-api-access-8t94f\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739291 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739235 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739359 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739336 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739413 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739371 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8242e4-ae99-448e-a660-a01fe5076393-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739464 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739417 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739464 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739446 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739467 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcz7\" (UniqueName: \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-kube-api-access-pbcz7\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.739556 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739504 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739639 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739577 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.739639 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739621 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.739746 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739719 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739801 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739785 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.739856 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739788 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.739899 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.739879 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8242e4-ae99-448e-a660-a01fe5076393-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.741601 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.741574 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.741807 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.741789 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8242e4-ae99-448e-a660-a01fe5076393-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.796254 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.796171 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.797593 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.797562 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t94f\" (UniqueName: \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-kube-api-access-8t94f\") pod \"maas-default-gateway-openshift-default-694db55c87-r6sd2\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.828721 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.828683 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:25.840793 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.840763 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcz7\" (UniqueName: \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-kube-api-access-pbcz7\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.840920 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.840827 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.840920 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.840901 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.841223 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.841198 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.841322 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.841288 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cce801-630f-4f59-8054-5bee59bde387-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.841381 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.841347 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.841437 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.841426 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.841492 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.841426 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.841545 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.841503 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.841603 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.841570 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cce801-630f-4f59-8054-5bee59bde387-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.841664 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.841649 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.841914 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.841892 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.842016 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.841996 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cce801-630f-4f59-8054-5bee59bde387-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.842092 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.842071 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.844172 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.844150 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cce801-630f-4f59-8054-5bee59bde387-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.844172 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.844161 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.883722 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.883280 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:59:25.883722 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.883342 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:59:25.897094 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.897068 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcz7\" (UniqueName: \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-kube-api-access-pbcz7\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.898520 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.898474 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-c7s57\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:25.902453 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:25.902365 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:26.288264 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.288229 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2"] Oct 30 21:59:26.289650 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:59:26.289626 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8242e4_ae99_448e_a660_a01fe5076393.slice/crio-ec88c127df6e3b17aed6ba09cddaf68dff6e502425d6223ea6e5b95a78685c50 WatchSource:0}: Error finding container ec88c127df6e3b17aed6ba09cddaf68dff6e502425d6223ea6e5b95a78685c50: Status 404 returned error can't find the container with id ec88c127df6e3b17aed6ba09cddaf68dff6e502425d6223ea6e5b95a78685c50 Oct 30 21:59:26.291987 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.291929 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.292095 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.292006 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.292095 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.292060 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.336860 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.336831 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57"] Oct 30 21:59:26.338253 ip-10-0-141-104 kubenswrapper[4859]: W1030 21:59:26.338229 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cce801_630f_4f59_8054_5bee59bde387.slice/crio-17d5ebdfb1bf0cba3da07071d1f4f5e582f4e2d9dc35239aa975a80401c9ede2 WatchSource:0}: Error finding container 17d5ebdfb1bf0cba3da07071d1f4f5e582f4e2d9dc35239aa975a80401c9ede2: Status 404 returned error can't find the container with id 17d5ebdfb1bf0cba3da07071d1f4f5e582f4e2d9dc35239aa975a80401c9ede2 Oct 30 21:59:26.340471 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.340444 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.340531 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.340507 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.340567 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.340536 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.414578 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.414171 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" event={"ID":"b6cce801-630f-4f59-8054-5bee59bde387","Type":"ContainerStarted","Data":"17d5ebdfb1bf0cba3da07071d1f4f5e582f4e2d9dc35239aa975a80401c9ede2"} Oct 30 21:59:26.416153 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.416126 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" event={"ID":"8c8242e4-ae99-448e-a660-a01fe5076393","Type":"ContainerStarted","Data":"b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc"} Oct 30 21:59:26.416285 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.416177 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" event={"ID":"8c8242e4-ae99-448e-a660-a01fe5076393","Type":"ContainerStarted","Data":"ec88c127df6e3b17aed6ba09cddaf68dff6e502425d6223ea6e5b95a78685c50"} Oct 30 21:59:26.534790 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.534487 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podStartSLOduration=1.534468626 podStartE2EDuration="1.534468626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:26.533466501 +0000 UTC m=+1023.906093251" watchObservedRunningTime="2026-10-30 21:59:26.534468626 +0000 UTC m=+1023.907095373" Oct 30 21:59:26.829408 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.829365 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:26.883967 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.882719 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:59:26.883967 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:26.882793 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:59:27.434367 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:27.434330 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" event={"ID":"b6cce801-630f-4f59-8054-5bee59bde387","Type":"ContainerStarted","Data":"b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5"} Oct 30 21:59:27.831120 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:27.831033 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6sd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:27.831120 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:27.831102 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:27.882392 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:27.882350 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:59:27.882634 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:27.882602 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:59:27.908625 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:27.908589 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:27.909322 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:27.909299 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:59:27.909482 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:27.909457 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 21:59:28.836343 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:28.836298 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6sd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:28.836579 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:28.836553 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:28.917970 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:28.915285 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:28.917970 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:28.916636 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:28.917970 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:28.917048 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:59:28.917970 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:28.917103 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 21:59:28.925497 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:28.923142 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 21:59:29.066216 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:29.066156 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podStartSLOduration=4.066139405 podStartE2EDuration="4.066139405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:27.558214737 +0000 UTC m=+1024.930841482" watchObservedRunningTime="2026-10-30 21:59:29.066139405 +0000 UTC m=+1026.438766150" Oct 30 21:59:29.830965 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:29.829477 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6sd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:29.830965 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:29.829529 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:29.903972 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:29.903204 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:59:29.903972 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:29.903262 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 21:59:30.829986 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:30.829765 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6sd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:30.829986 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:30.829824 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:30.904967 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:30.904304 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:59:30.904967 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:30.904364 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 21:59:31.829406 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:31.829373 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6sd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:31.829578 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:31.829429 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:31.907575 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:31.907528 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:59:31.908250 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:31.908183 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 21:59:32.829290 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:32.829244 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6sd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:32.829525 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:32.829322 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:32.902995 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:32.902959 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:59:32.903184 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:32.903018 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 21:59:33.829966 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:33.829919 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6sd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:33.830572 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:33.829997 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:33.903615 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:33.903580 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:59:33.903802 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:33.903650 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 21:59:35.115223 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:35.115144 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:35.116019 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:35.115995 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:35.117322 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:35.117298 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 21:59:35.128676 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:35.128650 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:35.129060 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:35.129043 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:35.129826 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:35.129806 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 21:59:36.421567 ip-10-0-141-104 kubenswrapper[4859]: I1030 21:59:36.421531 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 21:59:40.041555 ip-10-0-141-104 kubenswrapper[4859]: E1030 21:59:40.041516 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:00:00.180371 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.180321 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz"] Oct 30 22:00:00.185686 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.185659 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz"] Oct 30 22:00:00.185831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.185809 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" Oct 30 22:00:00.189556 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.189529 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" Oct 30 22:00:00.237595 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.237565 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\"" Oct 30 22:00:00.277985 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.277901 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz\" (UID: \"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" Oct 30 22:00:00.278182 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.278097 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzkl\" (UniqueName: \"kubernetes.io/projected/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-kube-api-access-cjzkl\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz\" (UID: \"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" Oct 30 22:00:00.278297 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.278278 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e01490-e7c1-45ce-84d4-62902897da70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz\" (UID: \"c7e01490-e7c1-45ce-84d4-62902897da70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" Oct 30 22:00:00.278376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.278314 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wmz\" (UniqueName: \"kubernetes.io/projected/c7e01490-e7c1-45ce-84d4-62902897da70-kube-api-access-s5wmz\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz\" (UID: \"c7e01490-e7c1-45ce-84d4-62902897da70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" Oct 30 22:00:00.292189 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.291934 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz"] Oct 30 22:00:00.296657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.296624 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz"] Oct 30 22:00:00.379118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.379077 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e01490-e7c1-45ce-84d4-62902897da70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz\" (UID: \"c7e01490-e7c1-45ce-84d4-62902897da70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" Oct 30 22:00:00.379289 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.379125 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wmz\" (UniqueName: \"kubernetes.io/projected/c7e01490-e7c1-45ce-84d4-62902897da70-kube-api-access-s5wmz\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz\" (UID: \"c7e01490-e7c1-45ce-84d4-62902897da70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" Oct 30 22:00:00.379289 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.379155 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz\" (UID: \"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" Oct 30 22:00:00.379289 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.379218 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzkl\" (UniqueName: \"kubernetes.io/projected/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-kube-api-access-cjzkl\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz\" (UID: \"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" Oct 30 22:00:00.379878 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.379854 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e01490-e7c1-45ce-84d4-62902897da70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz\" (UID: \"c7e01490-e7c1-45ce-84d4-62902897da70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" Oct 30 22:00:00.379924 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.379856 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz\" (UID: \"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" Oct 30 22:00:00.439223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.439131 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wmz\" (UniqueName: \"kubernetes.io/projected/c7e01490-e7c1-45ce-84d4-62902897da70-kube-api-access-s5wmz\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz\" (UID: \"c7e01490-e7c1-45ce-84d4-62902897da70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" Oct 30 22:00:00.442820 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.442781 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzkl\" (UniqueName: \"kubernetes.io/projected/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-kube-api-access-cjzkl\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz\" (UID: \"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" Oct 30 22:00:00.502614 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.502579 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" Oct 30 22:00:00.509303 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.509277 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" Oct 30 22:00:00.884426 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.884392 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz"] Oct 30 22:00:00.885812 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:00:00.885792 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e01490_e7c1_45ce_84d4_62902897da70.slice/crio-70cae750deeecc481b1265589abbe3bdedc1d2dfa214b5fb7b32f54a71497f44 WatchSource:0}: Error finding container 70cae750deeecc481b1265589abbe3bdedc1d2dfa214b5fb7b32f54a71497f44: Status 404 returned error can't find the container with id 70cae750deeecc481b1265589abbe3bdedc1d2dfa214b5fb7b32f54a71497f44 Oct 30 22:00:00.931176 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:00.931152 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz"] Oct 30 22:00:00.932277 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:00:00.932258 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7491a01_8c2c_4ad1_a33c_e9c6a0bbd102.slice/crio-eecb9f3e51226ef9da4cde4de52573cbc70b3c326d25f55fd46903a895ec6ccb WatchSource:0}: Error finding container eecb9f3e51226ef9da4cde4de52573cbc70b3c326d25f55fd46903a895ec6ccb: Status 404 returned error can't find the container with id eecb9f3e51226ef9da4cde4de52573cbc70b3c326d25f55fd46903a895ec6ccb Oct 30 22:00:01.705260 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:01.704931 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" event={"ID":"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102","Type":"ContainerStarted","Data":"1cf28527ecabb953404c711e94a4b23d1d1ae031154f2d4bf632fd6fd812bcc9"} Oct 30 22:00:01.705260 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:01.705021 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" event={"ID":"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102","Type":"ContainerStarted","Data":"eecb9f3e51226ef9da4cde4de52573cbc70b3c326d25f55fd46903a895ec6ccb"} Oct 30 22:00:01.707026 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:01.706999 4859 generic.go:356] "Generic (PLEG): container finished" podID="c7e01490-e7c1-45ce-84d4-62902897da70" containerID="db5ef2daf53b5592e9b0ad5f1466847438c4f6680ee744b2508b668ca6d82fad" exitCode=60 Oct 30 22:00:01.707178 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:01.707050 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" event={"ID":"c7e01490-e7c1-45ce-84d4-62902897da70","Type":"ContainerDied","Data":"db5ef2daf53b5592e9b0ad5f1466847438c4f6680ee744b2508b668ca6d82fad"} Oct 30 22:00:01.707178 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:01.707086 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" event={"ID":"c7e01490-e7c1-45ce-84d4-62902897da70","Type":"ContainerStarted","Data":"70cae750deeecc481b1265589abbe3bdedc1d2dfa214b5fb7b32f54a71497f44"} Oct 30 22:00:01.707392 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:01.707374 4859 scope.go:122] "RemoveContainer" containerID="db5ef2daf53b5592e9b0ad5f1466847438c4f6680ee744b2508b668ca6d82fad" Oct 30 22:00:01.819049 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:01.818998 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" podStartSLOduration=1.818985023 podStartE2EDuration="1.818985023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.817014443 +0000 UTC m=+1059.189641190" watchObservedRunningTime="2026-10-30 22:00:01.818985023 +0000 UTC m=+1059.191611768" Oct 30 22:00:02.716015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:02.715986 4859 generic.go:356] "Generic (PLEG): container finished" podID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerID="1cf28527ecabb953404c711e94a4b23d1d1ae031154f2d4bf632fd6fd812bcc9" exitCode=60 Oct 30 22:00:02.716484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:02.716074 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" event={"ID":"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102","Type":"ContainerDied","Data":"1cf28527ecabb953404c711e94a4b23d1d1ae031154f2d4bf632fd6fd812bcc9"} Oct 30 22:00:02.716484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:02.716308 4859 scope.go:122] "RemoveContainer" containerID="1cf28527ecabb953404c711e94a4b23d1d1ae031154f2d4bf632fd6fd812bcc9" Oct 30 22:00:02.718193 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:02.718165 4859 generic.go:356] "Generic (PLEG): container finished" podID="c7e01490-e7c1-45ce-84d4-62902897da70" containerID="637501e323961f8349b968b97941c97d32347230139367d46ce223bee17319fe" exitCode=60 Oct 30 22:00:02.718293 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:02.718199 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" event={"ID":"c7e01490-e7c1-45ce-84d4-62902897da70","Type":"ContainerDied","Data":"637501e323961f8349b968b97941c97d32347230139367d46ce223bee17319fe"} Oct 30 22:00:02.718293 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:02.718234 4859 scope.go:122] "RemoveContainer" containerID="db5ef2daf53b5592e9b0ad5f1466847438c4f6680ee744b2508b668ca6d82fad" Oct 30 22:00:02.718466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:02.718450 4859 scope.go:122] "RemoveContainer" containerID="637501e323961f8349b968b97941c97d32347230139367d46ce223bee17319fe" Oct 30 22:00:02.719037 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:02.719004 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz_odh-ai-gateway-infra(c7e01490-e7c1-45ce-84d4-62902897da70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" Oct 30 22:00:03.726264 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:03.726187 4859 scope.go:122] "RemoveContainer" containerID="637501e323961f8349b968b97941c97d32347230139367d46ce223bee17319fe" Oct 30 22:00:03.726630 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:03.726460 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz_odh-ai-gateway-infra(c7e01490-e7c1-45ce-84d4-62902897da70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" Oct 30 22:00:03.728041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:03.728019 4859 generic.go:356] "Generic (PLEG): container finished" podID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerID="7df83f0fd42f720f5f5cc8ca42f5db9e7f88d9ddc109efb284f9b1ac065ab895" exitCode=60 Oct 30 22:00:03.728155 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:03.728105 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" event={"ID":"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102","Type":"ContainerDied","Data":"7df83f0fd42f720f5f5cc8ca42f5db9e7f88d9ddc109efb284f9b1ac065ab895"} Oct 30 22:00:03.728202 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:03.728170 4859 scope.go:122] "RemoveContainer" containerID="1cf28527ecabb953404c711e94a4b23d1d1ae031154f2d4bf632fd6fd812bcc9" Oct 30 22:00:03.728335 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:03.728305 4859 scope.go:122] "RemoveContainer" containerID="7df83f0fd42f720f5f5cc8ca42f5db9e7f88d9ddc109efb284f9b1ac065ab895" Oct 30 22:00:03.728545 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:03.728527 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz_odh-ai-gateway-infra(d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" Oct 30 22:00:04.738692 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:04.738650 4859 scope.go:122] "RemoveContainer" containerID="7df83f0fd42f720f5f5cc8ca42f5db9e7f88d9ddc109efb284f9b1ac065ab895" Oct 30 22:00:04.739232 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:04.738925 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz_odh-ai-gateway-infra(d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" Oct 30 22:00:08.171018 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.170978 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:00:08.558136 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.558041 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj"] Oct 30 22:00:08.564664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.564632 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.581202 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.581166 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-home\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.581389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.581215 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-tmp-dir\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.581389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.581356 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5de8966-9960-46fb-8704-ae704cc77a6e-tls-certs\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.581514 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.581485 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4pg\" (UniqueName: \"kubernetes.io/projected/a5de8966-9960-46fb-8704-ae704cc77a6e-kube-api-access-qj4pg\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.581567 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.581517 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-model-cache\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.581645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.581629 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-dshm\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.666824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.666769 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e90e35\"/\"default-dockercfg-rxsvw\"" Oct 30 22:00:08.666824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.666778 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-e90e35\"/\"kube-root-ca.crt\"" Oct 30 22:00:08.666824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.666812 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-e90e35\"/\"openshift-service-ca.crt\"" Oct 30 22:00:08.667161 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.666870 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e90e35\"/\"distinct-w3-e90e35-kserve-self-signed-certs\"" Oct 30 22:00:08.681141 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.681109 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj"] Oct 30 22:00:08.682384 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.682356 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-dshm\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.682509 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.682427 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-home\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.682509 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.682455 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-tmp-dir\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.682509 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.682490 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5de8966-9960-46fb-8704-ae704cc77a6e-tls-certs\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.682670 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.682602 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4pg\" (UniqueName: \"kubernetes.io/projected/a5de8966-9960-46fb-8704-ae704cc77a6e-kube-api-access-qj4pg\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.682670 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.682629 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-model-cache\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.682984 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.682905 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-home\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.683159 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.682963 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-tmp-dir\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.683350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.683060 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-model-cache\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.684883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.684860 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-dshm\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.685640 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.685617 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5de8966-9960-46fb-8704-ae704cc77a6e-tls-certs\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.741898 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.741864 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4pg\" (UniqueName: \"kubernetes.io/projected/a5de8966-9960-46fb-8704-ae704cc77a6e-kube-api-access-qj4pg\") pod \"distinct-w3-e90e35-kserve-855889c455-dhnsj\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:08.877919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:08.877885 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:09.327799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:09.327762 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj"] Oct 30 22:00:09.330133 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:00:09.330101 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5de8966_9960_46fb_8704_ae704cc77a6e.slice/crio-21fe06f70c69b6c0a37e66c15770c47ce3be9c37ed57efab77d98b2ee67d5a64 WatchSource:0}: Error finding container 21fe06f70c69b6c0a37e66c15770c47ce3be9c37ed57efab77d98b2ee67d5a64: Status 404 returned error can't find the container with id 21fe06f70c69b6c0a37e66c15770c47ce3be9c37ed57efab77d98b2ee67d5a64 Oct 30 22:00:09.774416 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:09.774380 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" event={"ID":"a5de8966-9960-46fb-8704-ae704cc77a6e","Type":"ContainerStarted","Data":"51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b"} Oct 30 22:00:09.774599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:09.774425 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" event={"ID":"a5de8966-9960-46fb-8704-ae704cc77a6e","Type":"ContainerStarted","Data":"21fe06f70c69b6c0a37e66c15770c47ce3be9c37ed57efab77d98b2ee67d5a64"} Oct 30 22:00:09.877921 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:09.877881 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:00:09.878098 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:09.877983 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:09.888150 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:09.888102 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podStartSLOduration=1.888088448 podStartE2EDuration="1.888088448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:09.887341335 +0000 UTC m=+1067.259968082" watchObservedRunningTime="2026-10-30 22:00:09.888088448 +0000 UTC m=+1067.260715194" Oct 30 22:00:15.734132 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:15.734097 4859 scope.go:122] "RemoveContainer" containerID="637501e323961f8349b968b97941c97d32347230139367d46ce223bee17319fe" Oct 30 22:00:15.734596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:15.734207 4859 scope.go:122] "RemoveContainer" containerID="7df83f0fd42f720f5f5cc8ca42f5db9e7f88d9ddc109efb284f9b1ac065ab895" Oct 30 22:00:16.821520 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:16.821435 4859 generic.go:356] "Generic (PLEG): container finished" podID="c7e01490-e7c1-45ce-84d4-62902897da70" containerID="56f7d494b785195d4eeb7e00c5a26cad5a81d47e6dc3b890e24e7d0a27cd5862" exitCode=60 Oct 30 22:00:16.821520 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:16.821507 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" event={"ID":"c7e01490-e7c1-45ce-84d4-62902897da70","Type":"ContainerDied","Data":"56f7d494b785195d4eeb7e00c5a26cad5a81d47e6dc3b890e24e7d0a27cd5862"} Oct 30 22:00:16.822006 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:16.821567 4859 scope.go:122] "RemoveContainer" containerID="637501e323961f8349b968b97941c97d32347230139367d46ce223bee17319fe" Oct 30 22:00:16.822006 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:16.821896 4859 scope.go:122] "RemoveContainer" containerID="56f7d494b785195d4eeb7e00c5a26cad5a81d47e6dc3b890e24e7d0a27cd5862" Oct 30 22:00:16.822223 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:16.822197 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz_odh-ai-gateway-infra(c7e01490-e7c1-45ce-84d4-62902897da70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" Oct 30 22:00:16.823831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:16.823809 4859 generic.go:356] "Generic (PLEG): container finished" podID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerID="893004ec39c0d1a3906f92e6b92b3c7eb95c91b5503526cc6b0b0b2106aa32ed" exitCode=60 Oct 30 22:00:16.823926 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:16.823851 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" event={"ID":"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102","Type":"ContainerDied","Data":"893004ec39c0d1a3906f92e6b92b3c7eb95c91b5503526cc6b0b0b2106aa32ed"} Oct 30 22:00:16.824231 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:16.824218 4859 scope.go:122] "RemoveContainer" containerID="893004ec39c0d1a3906f92e6b92b3c7eb95c91b5503526cc6b0b0b2106aa32ed" Oct 30 22:00:16.824477 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:16.824458 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz_odh-ai-gateway-infra(d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" Oct 30 22:00:16.834450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:16.834429 4859 scope.go:122] "RemoveContainer" containerID="7df83f0fd42f720f5f5cc8ca42f5db9e7f88d9ddc109efb284f9b1ac065ab895" Oct 30 22:00:17.949025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:17.948994 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz"] Oct 30 22:00:18.062249 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.062200 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz"] Oct 30 22:00:18.103841 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.103802 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" Oct 30 22:00:18.204528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.204456 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e01490-e7c1-45ce-84d4-62902897da70-kube-api-access-s5wmz\" (UniqueName: \"kubernetes.io/projected/c7e01490-e7c1-45ce-84d4-62902897da70-kube-api-access-s5wmz\") pod \"c7e01490-e7c1-45ce-84d4-62902897da70\" (UID: \"c7e01490-e7c1-45ce-84d4-62902897da70\") " Oct 30 22:00:18.204663 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.204592 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e01490-e7c1-45ce-84d4-62902897da70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e01490-e7c1-45ce-84d4-62902897da70-maas-api-service-ca\") pod \"c7e01490-e7c1-45ce-84d4-62902897da70\" (UID: \"c7e01490-e7c1-45ce-84d4-62902897da70\") " Oct 30 22:00:18.205072 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.204917 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e01490-e7c1-45ce-84d4-62902897da70-maas-api-service-ca" pod "c7e01490-e7c1-45ce-84d4-62902897da70" (UID: "c7e01490-e7c1-45ce-84d4-62902897da70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.205140 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.205077 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e01490-e7c1-45ce-84d4-62902897da70-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.206742 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.206717 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e01490-e7c1-45ce-84d4-62902897da70-kube-api-access-s5wmz" pod "c7e01490-e7c1-45ce-84d4-62902897da70" (UID: "c7e01490-e7c1-45ce-84d4-62902897da70"). InnerVolumeSpecName "kube-api-access-s5wmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.223847 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.223825 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" Oct 30 22:00:18.306463 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.306425 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-maas-api-service-ca\") pod \"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102\" (UID: \"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102\") " Oct 30 22:00:18.306652 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.306564 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-kube-api-access-cjzkl\" (UniqueName: \"kubernetes.io/projected/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-kube-api-access-cjzkl\") pod \"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102\" (UID: \"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102\") " Oct 30 22:00:18.306880 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.306847 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-maas-api-service-ca" pod "d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" (UID: "d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.306969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.306885 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5wmz\" (UniqueName: \"kubernetes.io/projected/c7e01490-e7c1-45ce-84d4-62902897da70-kube-api-access-s5wmz\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.308636 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.308610 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-kube-api-access-cjzkl" pod "d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" (UID: "d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102"). InnerVolumeSpecName "kube-api-access-cjzkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.408111 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.408075 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.408111 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.408112 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjzkl\" (UniqueName: \"kubernetes.io/projected/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102-kube-api-access-cjzkl\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.846435 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.846404 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" Oct 30 22:00:18.846642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.846434 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz" event={"ID":"c7e01490-e7c1-45ce-84d4-62902897da70","Type":"ContainerDied","Data":"70cae750deeecc481b1265589abbe3bdedc1d2dfa214b5fb7b32f54a71497f44"} Oct 30 22:00:18.846642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.846496 4859 scope.go:122] "RemoveContainer" containerID="56f7d494b785195d4eeb7e00c5a26cad5a81d47e6dc3b890e24e7d0a27cd5862" Oct 30 22:00:18.849621 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.849563 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" Oct 30 22:00:18.849735 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.849613 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz" event={"ID":"d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102","Type":"ContainerDied","Data":"eecb9f3e51226ef9da4cde4de52573cbc70b3c326d25f55fd46903a895ec6ccb"} Oct 30 22:00:18.857424 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.857405 4859 scope.go:122] "RemoveContainer" containerID="893004ec39c0d1a3906f92e6b92b3c7eb95c91b5503526cc6b0b0b2106aa32ed" Oct 30 22:00:18.878063 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.878037 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:18.953764 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:18.953732 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:19.019559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:19.019467 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz"] Oct 30 22:00:19.022913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:19.022883 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889960-dhcdz"] Oct 30 22:00:19.132234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:19.132204 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz"] Oct 30 22:00:19.138209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:19.138149 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889960-nrvxz"] Oct 30 22:00:19.871797 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:19.871766 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:00:20.743690 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:20.743656 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" path="/var/lib/kubelet/pods/c7e01490-e7c1-45ce-84d4-62902897da70/volumes" Oct 30 22:00:20.744147 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:20.744129 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" path="/var/lib/kubelet/pods/d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102/volumes" Oct 30 22:00:28.887431 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:28.887398 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:00:39.878718 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:39.878670 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:00:48.778440 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:00:48.778401 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:00:48.872812 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:48.872771 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-501bd36994fb: secret "agc-cred-501bd36994fb" not found Oct 30 22:00:48.872812 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:48.872808 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5577b47559-4ll85: secret "agc-cred-501bd36994fb" not found Oct 30 22:00:48.873092 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:48.872885 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials podName:92e4f3dd-f33c-482b-add4-bde12575edd0 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:49.372864118 +0000 UTC m=+1106.745490856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials") pod "payload-processing-external-model-5577b47559-4ll85" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0") : secret "agc-cred-501bd36994fb" not found Oct 30 22:00:49.378850 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:49.378816 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-501bd36994fb: secret "agc-cred-501bd36994fb" not found Oct 30 22:00:49.379113 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:49.379085 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5577b47559-4ll85: secret "agc-cred-501bd36994fb" not found Oct 30 22:00:49.379215 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:49.379207 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials podName:92e4f3dd-f33c-482b-add4-bde12575edd0 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:50.379185414 +0000 UTC m=+1107.751812152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials") pod "payload-processing-external-model-5577b47559-4ll85" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0") : secret "agc-cred-501bd36994fb" not found Oct 30 22:00:50.391103 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:50.391057 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-501bd36994fb: secret "agc-cred-501bd36994fb" not found Oct 30 22:00:50.391103 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:50.391108 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5577b47559-4ll85: secret "agc-cred-501bd36994fb" not found Oct 30 22:00:50.391561 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:50.391183 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials podName:92e4f3dd-f33c-482b-add4-bde12575edd0 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:52.391162062 +0000 UTC m=+1109.763788810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials") pod "payload-processing-external-model-5577b47559-4ll85" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0") : secret "agc-cred-501bd36994fb" not found Oct 30 22:00:52.416994 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:52.416959 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-501bd36994fb: secret "agc-cred-501bd36994fb" not found Oct 30 22:00:52.416994 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:52.416991 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5577b47559-4ll85: secret "agc-cred-501bd36994fb" not found Oct 30 22:00:52.417416 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:52.417046 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials podName:92e4f3dd-f33c-482b-add4-bde12575edd0 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:56.417029205 +0000 UTC m=+1113.789655941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials") pod "payload-processing-external-model-5577b47559-4ll85" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0") : secret "agc-cred-501bd36994fb" not found Oct 30 22:00:56.464838 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:56.464801 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-501bd36994fb: secret "agc-cred-501bd36994fb" not found Oct 30 22:00:56.464838 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:56.464837 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5577b47559-4ll85: secret "agc-cred-501bd36994fb" not found Oct 30 22:00:56.465284 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:00:56.464897 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials podName:92e4f3dd-f33c-482b-add4-bde12575edd0 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:04.464879395 +0000 UTC m=+1121.837506121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials") pod "payload-processing-external-model-5577b47559-4ll85" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0") : secret "agc-cred-501bd36994fb" not found Oct 30 22:01:03.639336 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.639301 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv"] Oct 30 22:01:03.639991 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.639962 4859 cpu_manager.go:414] "containerMap: removing container" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" containerName="cleanup" Oct 30 22:01:03.639991 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.639989 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" containerName="cleanup" Oct 30 22:01:03.640169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640008 4859 cpu_manager.go:414] "containerMap: removing container" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerName="cleanup" Oct 30 22:01:03.640169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640014 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerName="cleanup" Oct 30 22:01:03.640169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640027 4859 cpu_manager.go:414] "containerMap: removing container" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" containerName="cleanup" Oct 30 22:01:03.640169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640032 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" containerName="cleanup" Oct 30 22:01:03.640169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640047 4859 cpu_manager.go:414] "containerMap: removing container" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerName="cleanup" Oct 30 22:01:03.640169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640054 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerName="cleanup" Oct 30 22:01:03.640169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640174 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerName="cleanup" Oct 30 22:01:03.640387 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640183 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" containerName="cleanup" Oct 30 22:01:03.640387 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640190 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerName="cleanup" Oct 30 22:01:03.640387 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640196 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" containerName="cleanup" Oct 30 22:01:03.640387 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640202 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerName="cleanup" Oct 30 22:01:03.640387 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.640209 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" containerName="cleanup" Oct 30 22:01:03.644507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.644490 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.654795 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.654767 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-provider-credentials\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.654910 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.654804 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvqv\" (UniqueName: \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-kube-api-access-9lvqv\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.654910 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.654848 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-routing-overlay-volume\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.655064 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.655037 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-plugins-config-volume\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.748908 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.748842 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv"] Oct 30 22:01:03.756314 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.756284 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-provider-credentials\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.756460 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.756336 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvqv\" (UniqueName: \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-kube-api-access-9lvqv\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.756522 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.756461 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-routing-overlay-volume\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.756522 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.756504 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-plugins-config-volume\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.757337 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.757309 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-routing-overlay-volume\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.757580 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.757554 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-plugins-config-volume\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.759010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.758989 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-provider-credentials\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.815408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.815371 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvqv\" (UniqueName: \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-kube-api-access-9lvqv\") pod \"payload-processing-external-model-66f8fcd7fb-j5swv\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:03.957969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:03.957859 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:04.332048 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:04.332015 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv"] Oct 30 22:01:04.333046 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:01:04.333019 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0984700_dd94_491b_a149_cf2e55b6f770.slice/crio-f2ef8fb23d0058a217affa211515e1b63acf647b2f6d6f9963c8e7a574b00195 WatchSource:0}: Error finding container f2ef8fb23d0058a217affa211515e1b63acf647b2f6d6f9963c8e7a574b00195: Status 404 returned error can't find the container with id f2ef8fb23d0058a217affa211515e1b63acf647b2f6d6f9963c8e7a574b00195 Oct 30 22:01:04.465171 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:04.465136 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7513640d12da: secret "agc-cred-7513640d12da" not found Oct 30 22:01:04.465171 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:04.465164 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5577b47559-4ll85: secret "agc-cred-7513640d12da" not found Oct 30 22:01:04.465396 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:04.465228 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials podName:92e4f3dd-f33c-482b-add4-bde12575edd0 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:20.46520815 +0000 UTC m=+1137.837834887 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials") pod "payload-processing-external-model-5577b47559-4ll85" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0") : secret "agc-cred-7513640d12da" not found Oct 30 22:01:05.180182 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:05.180100 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" event={"ID":"d0984700-dd94-491b-a149-cf2e55b6f770","Type":"ContainerStarted","Data":"a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e"} Oct 30 22:01:05.180182 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:05.180154 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" event={"ID":"d0984700-dd94-491b-a149-cf2e55b6f770","Type":"ContainerStarted","Data":"f2ef8fb23d0058a217affa211515e1b63acf647b2f6d6f9963c8e7a574b00195"} Oct 30 22:01:05.180182 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:05.180180 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:01:05.301038 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:05.300973 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" podStartSLOduration=2.300955863 podStartE2EDuration="2.300955863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:05.297560901 +0000 UTC m=+1122.670187648" watchObservedRunningTime="2026-10-30 22:01:05.300955863 +0000 UTC m=+1122.673582602" Oct 30 22:01:09.982151 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:09.982113 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:01:16.191408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:16.191369 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:01:20.477113 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:20.477078 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7513640d12da: secret "agc-cred-7513640d12da" not found Oct 30 22:01:20.477113 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:20.477105 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5577b47559-4ll85: secret "agc-cred-7513640d12da" not found Oct 30 22:01:20.477651 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:20.477168 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials podName:92e4f3dd-f33c-482b-add4-bde12575edd0 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:52.477150797 +0000 UTC m=+1169.849777523 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials") pod "payload-processing-external-model-5577b47559-4ll85" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0") : secret "agc-cred-7513640d12da" not found Oct 30 22:01:23.066702 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:23.066669 4859 scope.go:122] "RemoveContainer" containerID="ec9fe182740de58a116375e63a2c6ab8e987960fbdd56f47a83465d2a99a231e" Oct 30 22:01:23.076912 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:23.076889 4859 scope.go:122] "RemoveContainer" containerID="2b1b355b923aac167944b3210b71b55261f92a18a27e1b25adbec0d2a212c1b7" Oct 30 22:01:23.516181 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:23.516148 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:01:26.189902 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:26.189854 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:01:36.189594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:36.189554 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:01:39.881187 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:39.881141 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:01:45.749374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:45.749337 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:01:46.194968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:46.194045 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:01:52.569322 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:52.569283 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7513640d12da: secret "agc-cred-7513640d12da" not found Oct 30 22:01:52.569322 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:52.569311 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5577b47559-4ll85: secret "agc-cred-7513640d12da" not found Oct 30 22:01:52.569964 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:01:52.569376 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials podName:92e4f3dd-f33c-482b-add4-bde12575edd0 nodeName:}" failed. No retries permitted until 2026-10-30 22:02:56.56936046 +0000 UTC m=+1233.941987186 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials") pod "payload-processing-external-model-5577b47559-4ll85" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0") : secret "agc-cred-7513640d12da" not found Oct 30 22:01:56.190031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:01:56.189991 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:06.190345 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:06.190306 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:09.766403 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.766366 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw"] Oct 30 22:02:09.767138 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.767117 4859 cpu_manager.go:414] "containerMap: removing container" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" containerName="cleanup" Oct 30 22:02:09.767138 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.767135 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e01490-e7c1-45ce-84d4-62902897da70" containerName="cleanup" Oct 30 22:02:09.767312 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.767150 4859 cpu_manager.go:414] "containerMap: removing container" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerName="cleanup" Oct 30 22:02:09.767312 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.767155 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7491a01-8c2c-4ad1-a33c-e9c6a0bbd102" containerName="cleanup" Oct 30 22:02:09.770986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.770968 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.877606 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.877564 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.877795 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.877643 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.880486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.880451 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.880710 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.880687 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.880911 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.880888 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.881103 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.881084 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.881383 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.881361 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.881564 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.881545 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.881647 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.881587 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdkl\" (UniqueName: \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-kube-api-access-dqdkl\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.896277 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.896243 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw"] Oct 30 22:02:09.898353 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:02:09.898320 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:02:09.983484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.983448 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.983484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.983488 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.983747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.983505 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdkl\" (UniqueName: \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-kube-api-access-dqdkl\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.983747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.983585 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.983747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.983632 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.983747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.983658 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.983747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.983678 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.983747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.983725 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.984154 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.983771 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.984154 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.983973 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.984154 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.984048 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.984323 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.984272 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.984529 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.984503 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.984529 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.984519 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.986067 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.986047 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:09.986390 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:09.986366 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:10.048886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.048804 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:10.048886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.048866 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdkl\" (UniqueName: \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-kube-api-access-dqdkl\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:10.085133 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.085091 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:10.514199 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.514151 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw"] Oct 30 22:02:10.515388 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:02:10.515360 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748bacd4_dd89_4c59_8be2_1a8bae565bb7.slice/crio-85c3e712c681a8cc61b2841aa5a76661a7ef6ed8bd86c658109d41867e86ece8 WatchSource:0}: Error finding container 85c3e712c681a8cc61b2841aa5a76661a7ef6ed8bd86c658109d41867e86ece8: Status 404 returned error can't find the container with id 85c3e712c681a8cc61b2841aa5a76661a7ef6ed8bd86c658109d41867e86ece8 Oct 30 22:02:10.517863 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.517822 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:10.517988 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.517896 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:10.517988 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.517929 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:10.672483 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.672448 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv"] Oct 30 22:02:10.673928 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.673892 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:10.685958 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.685909 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" event={"ID":"748bacd4-dd89-4c59-8be2-1a8bae565bb7","Type":"ContainerStarted","Data":"c5cd7522cfb3a2e1d7e16c9e94ab944a33ea0f36aa85953d8286fd115acfcbed"} Oct 30 22:02:10.686150 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.685998 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" event={"ID":"748bacd4-dd89-4c59-8be2-1a8bae565bb7","Type":"ContainerStarted","Data":"85c3e712c681a8cc61b2841aa5a76661a7ef6ed8bd86c658109d41867e86ece8"} Oct 30 22:02:10.699831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.699794 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-747d767c44-v9lrd"] Oct 30 22:02:10.704250 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.704225 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.758618 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.758586 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-32223ebd7979\"" Oct 30 22:02:10.758905 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.758808 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6ede377aee21\"" Oct 30 22:02:10.764368 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.764290 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-747d767c44-v9lrd"] Oct 30 22:02:10.794989 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.794899 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-plugins-config-volume\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.795498 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.795033 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-routing-overlay-volume\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.796992 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.796635 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.796992 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.796843 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trmhw\" (UniqueName: \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-kube-api-access-trmhw\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.898285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.898245 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.898480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.898325 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trmhw\" (UniqueName: \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-kube-api-access-trmhw\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.898480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.898368 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-plugins-config-volume\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.898480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.898395 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-routing-overlay-volume\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.899144 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.899118 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-routing-overlay-volume\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.899257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.899202 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-plugins-config-volume\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.901011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.900984 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:10.904787 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.904740 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podStartSLOduration=1.904726565 podStartE2EDuration="1.904726565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:10.901398858 +0000 UTC m=+1188.274025615" watchObservedRunningTime="2026-10-30 22:02:10.904726565 +0000 UTC m=+1188.277353311" Oct 30 22:02:10.956398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:10.956369 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trmhw\" (UniqueName: \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-kube-api-access-trmhw\") pod \"payload-processing-external-model-747d767c44-v9lrd\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:11.020996 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:11.019262 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:11.086968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:11.086335 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:11.092134 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:11.092091 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:02:11.092328 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:11.092160 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:02:11.423618 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:11.423582 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-747d767c44-v9lrd"] Oct 30 22:02:11.424816 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:02:11.424790 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812d6776_ca57_42f7_afe1_27f4fcc03cae.slice/crio-7ca0d3b68355628840b923c4a25264d91c61b0a71cce500f134eb59901737389 WatchSource:0}: Error finding container 7ca0d3b68355628840b923c4a25264d91c61b0a71cce500f134eb59901737389: Status 404 returned error can't find the container with id 7ca0d3b68355628840b923c4a25264d91c61b0a71cce500f134eb59901737389 Oct 30 22:02:11.694267 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:11.694155 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" event={"ID":"812d6776-ca57-42f7-afe1-27f4fcc03cae","Type":"ContainerStarted","Data":"2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934"} Oct 30 22:02:11.694267 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:11.694227 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" event={"ID":"812d6776-ca57-42f7-afe1-27f4fcc03cae","Type":"ContainerStarted","Data":"7ca0d3b68355628840b923c4a25264d91c61b0a71cce500f134eb59901737389"} Oct 30 22:02:11.694495 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:11.694373 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:11.812761 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:11.812710 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" podStartSLOduration=1.812690098 podStartE2EDuration="1.812690098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:11.812401298 +0000 UTC m=+1189.185028045" watchObservedRunningTime="2026-10-30 22:02:11.812690098 +0000 UTC m=+1189.185316844" Oct 30 22:02:12.086520 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:12.086424 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:02:12.086520 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:12.086501 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:02:13.089967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:13.089386 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:02:13.089967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:13.089455 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:02:14.086968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:14.086384 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:02:14.086968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:14.086463 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:02:15.086384 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:15.086345 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:02:15.087052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:15.087009 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:02:15.676969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:15.675021 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" containerID="cri-o://a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e" gracePeriod=25 Oct 30 22:02:16.010694 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.010663 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:02:16.089980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.089291 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:02:16.089980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.089365 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:02:16.106188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.106157 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-plugins-config-volume\") pod \"d0984700-dd94-491b-a149-cf2e55b6f770\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " Oct 30 22:02:16.106358 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.106208 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-kube-api-access-9lvqv\" (UniqueName: \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-kube-api-access-9lvqv\") pod \"d0984700-dd94-491b-a149-cf2e55b6f770\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " Oct 30 22:02:16.106358 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.106276 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-routing-overlay-volume\") pod \"d0984700-dd94-491b-a149-cf2e55b6f770\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " Oct 30 22:02:16.106358 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.106337 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-provider-credentials\") pod \"d0984700-dd94-491b-a149-cf2e55b6f770\" (UID: \"d0984700-dd94-491b-a149-cf2e55b6f770\") " Oct 30 22:02:16.106902 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.106652 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-plugins-config-volume" pod "d0984700-dd94-491b-a149-cf2e55b6f770" (UID: "d0984700-dd94-491b-a149-cf2e55b6f770"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:16.106902 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.106691 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-routing-overlay-volume" pod "d0984700-dd94-491b-a149-cf2e55b6f770" (UID: "d0984700-dd94-491b-a149-cf2e55b6f770"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:16.107114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.106937 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.107114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.106990 4859 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0984700-dd94-491b-a149-cf2e55b6f770-routing-overlay-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.109134 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.109114 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-kube-api-access-9lvqv" pod "d0984700-dd94-491b-a149-cf2e55b6f770" (UID: "d0984700-dd94-491b-a149-cf2e55b6f770"). InnerVolumeSpecName "kube-api-access-9lvqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:16.109199 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.109146 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-provider-credentials" pod "d0984700-dd94-491b-a149-cf2e55b6f770" (UID: "d0984700-dd94-491b-a149-cf2e55b6f770"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:16.216083 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.213843 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lvqv\" (UniqueName: \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-kube-api-access-9lvqv\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.216083 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.213884 4859 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0984700-dd94-491b-a149-cf2e55b6f770-provider-credentials\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.733709 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.733672 4859 generic.go:356] "Generic (PLEG): container finished" podID="d0984700-dd94-491b-a149-cf2e55b6f770" containerID="a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e" exitCode=0 Oct 30 22:02:16.733970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.733934 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" Oct 30 22:02:16.744187 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.744158 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" event={"ID":"d0984700-dd94-491b-a149-cf2e55b6f770","Type":"ContainerDied","Data":"a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e"} Oct 30 22:02:16.744313 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.744191 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv" event={"ID":"d0984700-dd94-491b-a149-cf2e55b6f770","Type":"ContainerDied","Data":"f2ef8fb23d0058a217affa211515e1b63acf647b2f6d6f9963c8e7a574b00195"} Oct 30 22:02:16.744313 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.744213 4859 scope.go:122] "RemoveContainer" containerID="a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e" Oct 30 22:02:16.759027 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.759004 4859 scope.go:122] "RemoveContainer" containerID="a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e" Oct 30 22:02:16.759394 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:02:16.759367 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e\": container with ID starting with a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e not found: ID does not exist" containerID="a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e" Oct 30 22:02:16.759446 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.759404 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e"} err="failed to get container status \"a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e\": rpc error: code = NotFound desc = could not find container \"a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e\": container with ID starting with a9b90ef6762afa7bf54edb63a296c575024a3ce6970b0c2d0a206e3d801eef7e not found: ID does not exist" Oct 30 22:02:16.899045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.898964 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv"] Oct 30 22:02:16.958636 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:16.958600 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-j5swv"] Oct 30 22:02:17.070359 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:17.068226 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:02:17.085967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:17.085558 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:02:17.085967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:17.085622 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:02:18.091043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:18.091010 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:18.091508 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:18.091442 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:18.092183 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:18.092166 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:02:18.745366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:18.745326 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" path="/var/lib/kubelet/pods/d0984700-dd94-491b-a149-cf2e55b6f770/volumes" Oct 30 22:02:21.705612 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.705581 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd"] Oct 30 22:02:21.706160 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.706146 4859 cpu_manager.go:414] "containerMap: removing container" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" Oct 30 22:02:21.706209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.706161 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" Oct 30 22:02:21.706275 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.706266 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0984700-dd94-491b-a149-cf2e55b6f770" containerName="payload-processing" Oct 30 22:02:21.711039 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.711021 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.771217 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.771190 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd"] Oct 30 22:02:21.843784 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.843753 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.843907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.843802 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.843907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.843864 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.843907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.843894 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.844050 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.844027 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dec6bd6-145b-4257-8077-4588a7e68960-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.844085 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.844060 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.844118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.844101 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx5fb\" (UniqueName: \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-kube-api-access-xx5fb\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.844283 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.844264 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.844337 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.844319 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dec6bd6-145b-4257-8077-4588a7e68960-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946052 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dec6bd6-145b-4257-8077-4588a7e68960-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946087 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946118 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx5fb\" (UniqueName: \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-kube-api-access-xx5fb\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946162 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946316 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946197 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dec6bd6-145b-4257-8077-4588a7e68960-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946316 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946256 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946316 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946292 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946470 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946319 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946523 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946468 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946661 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946625 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946676 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946743 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.946977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.946837 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.947094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.947068 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dec6bd6-145b-4257-8077-4588a7e68960-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.948400 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.948378 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:21.948654 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:21.948636 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dec6bd6-145b-4257-8077-4588a7e68960-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:22.003593 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.003537 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:22.003711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.003693 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx5fb\" (UniqueName: \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-kube-api-access-xx5fb\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:22.024619 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.024601 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:22.455712 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.455689 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd"] Oct 30 22:02:22.456477 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:02:22.456453 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dec6bd6_145b_4257_8077_4588a7e68960.slice/crio-64312dba91f55e43f36dc9450e70631c174f1cde22ed755a396cb1aa23852fac WatchSource:0}: Error finding container 64312dba91f55e43f36dc9450e70631c174f1cde22ed755a396cb1aa23852fac: Status 404 returned error can't find the container with id 64312dba91f55e43f36dc9450e70631c174f1cde22ed755a396cb1aa23852fac Oct 30 22:02:22.458777 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.458735 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:22.458868 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.458803 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:22.458868 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.458842 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:22.707745 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.707671 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:02:22.795021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.794935 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" event={"ID":"0dec6bd6-145b-4257-8077-4588a7e68960","Type":"ContainerStarted","Data":"dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c"} Oct 30 22:02:22.795200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.795078 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" event={"ID":"0dec6bd6-145b-4257-8077-4588a7e68960","Type":"ContainerStarted","Data":"64312dba91f55e43f36dc9450e70631c174f1cde22ed755a396cb1aa23852fac"} Oct 30 22:02:22.857292 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.857253 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5577b47559-4ll85"] Oct 30 22:02:22.945615 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.945585 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:02:22.947031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.946996 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:02:22.960577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.960521 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:02:22.969501 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:22.969475 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:02:23.025397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:23.025362 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:23.029668 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:23.029151 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:23.029668 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:23.029212 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:02:23.039969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:23.039159 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podStartSLOduration=2.039138447 podStartE2EDuration="2.039138447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:23.034364493 +0000 UTC m=+1200.406991242" watchObservedRunningTime="2026-10-30 22:02:23.039138447 +0000 UTC m=+1200.411765194" Oct 30 22:02:24.025879 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:24.025845 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:24.026352 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:24.025896 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:02:25.025356 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:25.025321 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:25.025535 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:25.025374 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:02:26.026009 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:26.025976 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:26.026503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:26.026033 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:02:27.025404 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:27.025364 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:27.025617 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:27.025430 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:02:27.858230 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:27.858178 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" podUID="92e4f3dd-f33c-482b-add4-bde12575edd0" containerName="payload-processing" containerID="cri-o://e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1" gracePeriod=25 Oct 30 22:02:28.026019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.025984 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:28.026156 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.026050 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:02:28.185978 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.184109 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 22:02:28.241978 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.241936 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-kube-api-access-t49q9\" (UniqueName: \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-kube-api-access-t49q9\") pod \"92e4f3dd-f33c-482b-add4-bde12575edd0\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " Oct 30 22:02:28.242157 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.242140 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-plugins-config-volume\") pod \"92e4f3dd-f33c-482b-add4-bde12575edd0\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " Oct 30 22:02:28.242220 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.242208 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials\") pod \"92e4f3dd-f33c-482b-add4-bde12575edd0\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " Oct 30 22:02:28.242304 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.242288 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-routing-overlay-volume\") pod \"92e4f3dd-f33c-482b-add4-bde12575edd0\" (UID: \"92e4f3dd-f33c-482b-add4-bde12575edd0\") " Oct 30 22:02:28.242643 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.242604 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-routing-overlay-volume" pod "92e4f3dd-f33c-482b-add4-bde12575edd0" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:28.242813 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.242757 4859 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-routing-overlay-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:28.242864 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.242849 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-plugins-config-volume" pod "92e4f3dd-f33c-482b-add4-bde12575edd0" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:28.244270 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.244246 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials" pod "92e4f3dd-f33c-482b-add4-bde12575edd0" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:28.244376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.244311 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-kube-api-access-t49q9" pod "92e4f3dd-f33c-482b-add4-bde12575edd0" (UID: "92e4f3dd-f33c-482b-add4-bde12575edd0"). InnerVolumeSpecName "kube-api-access-t49q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:28.343588 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.343557 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t49q9\" (UniqueName: \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-kube-api-access-t49q9\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:28.343588 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.343585 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e4f3dd-f33c-482b-add4-bde12575edd0-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:28.343588 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.343595 4859 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92e4f3dd-f33c-482b-add4-bde12575edd0-provider-credentials\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:28.839362 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.839329 4859 generic.go:356] "Generic (PLEG): container finished" podID="92e4f3dd-f33c-482b-add4-bde12575edd0" containerID="e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1" exitCode=0 Oct 30 22:02:28.839536 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.839360 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" event={"ID":"92e4f3dd-f33c-482b-add4-bde12575edd0","Type":"ContainerDied","Data":"e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1"} Oct 30 22:02:28.839536 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.839392 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" Oct 30 22:02:28.839536 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.839426 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5577b47559-4ll85" event={"ID":"92e4f3dd-f33c-482b-add4-bde12575edd0","Type":"ContainerDied","Data":"e9df374f475ae52e6217759e62a69b948d3c6ccdc3c78a249646b0dead302438"} Oct 30 22:02:28.839536 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.839455 4859 scope.go:122] "RemoveContainer" containerID="e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1" Oct 30 22:02:28.859149 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.858896 4859 scope.go:122] "RemoveContainer" containerID="e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1" Oct 30 22:02:28.859467 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:02:28.859255 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1\": container with ID starting with e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1 not found: ID does not exist" containerID="e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1" Oct 30 22:02:28.859467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:28.859290 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1"} err="failed to get container status \"e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1\": rpc error: code = NotFound desc = could not find container \"e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1\": container with ID starting with e1bb40a9c90aa3ebd46276879f66586a84111799493c5b543c3fc18786505ba1 not found: ID does not exist" Oct 30 22:02:29.008000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:29.007903 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5577b47559-4ll85"] Oct 30 22:02:29.026081 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:29.026051 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:29.026193 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:29.026125 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:02:29.057374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:29.057347 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5577b47559-4ll85"] Oct 30 22:02:30.029710 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:30.029676 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:30.030241 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:30.030219 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:30.031046 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:30.031024 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:02:30.741240 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:30.741203 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e4f3dd-f33c-482b-add4-bde12575edd0" path="/var/lib/kubelet/pods/92e4f3dd-f33c-482b-add4-bde12575edd0/volumes" Oct 30 22:02:36.174826 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:36.174734 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:02:36.175583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:36.175544 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:02:39.877716 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:02:39.877681 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:02:50.246917 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:02:50.246883 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:03:06.783209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.783170 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57"] Oct 30 22:03:06.783799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.783738 4859 cpu_manager.go:414] "containerMap: removing container" podUID="92e4f3dd-f33c-482b-add4-bde12575edd0" containerName="payload-processing" Oct 30 22:03:06.783799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.783750 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e4f3dd-f33c-482b-add4-bde12575edd0" containerName="payload-processing" Oct 30 22:03:06.783924 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.783838 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e4f3dd-f33c-482b-add4-bde12575edd0" containerName="payload-processing" Oct 30 22:03:06.788503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.788482 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:06.895881 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.895845 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57"] Oct 30 22:03:06.929578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.929545 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:06.929743 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.929587 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tgz\" (UniqueName: \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-kube-api-access-96tgz\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:06.929743 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.929669 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:06.929831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.929754 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:06.929929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.929906 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:06.930063 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.930045 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:06.930157 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.930143 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:06.930192 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.930171 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:06.930256 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:06.930240 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.031332 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.031296 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.031493 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.031343 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.031493 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.031427 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.031582 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.031511 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.031636 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.031579 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.031636 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.031604 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.031729 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.031642 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.031729 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.031709 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.031815 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.031740 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tgz\" (UniqueName: \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-kube-api-access-96tgz\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.031870 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.031830 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.032120 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.032092 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.032236 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.032178 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.032325 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.032300 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.032554 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.032532 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.033971 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.033899 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.034158 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.034141 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.089175 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.089146 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.089285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.089270 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tgz\" (UniqueName: \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-kube-api-access-96tgz\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.106012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.105991 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:07.536466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.536441 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57"] Oct 30 22:03:07.536993 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:03:07.536964 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edd5bc5_174f_4554_9d7d_a5f0d9e1170e.slice/crio-5205d4a4def91d44a75221d546f40c809a796823b6923613f494bb02d5c266b1 WatchSource:0}: Error finding container 5205d4a4def91d44a75221d546f40c809a796823b6923613f494bb02d5c266b1: Status 404 returned error can't find the container with id 5205d4a4def91d44a75221d546f40c809a796823b6923613f494bb02d5c266b1 Oct 30 22:03:07.539062 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.539026 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:07.539168 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.539112 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:07.539168 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:07.539158 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:08.258017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:08.257981 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" event={"ID":"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e","Type":"ContainerStarted","Data":"a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9"} Oct 30 22:03:08.258434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:08.258041 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" event={"ID":"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e","Type":"ContainerStarted","Data":"5205d4a4def91d44a75221d546f40c809a796823b6923613f494bb02d5c266b1"} Oct 30 22:03:08.424196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:08.424145 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" podStartSLOduration=2.424129926 podStartE2EDuration="2.424129926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:08.422542052 +0000 UTC m=+1245.795168800" watchObservedRunningTime="2026-10-30 22:03:08.424129926 +0000 UTC m=+1245.796756751" Oct 30 22:03:09.106233 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:09.106193 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:09.107545 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:09.107517 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:03:09.107670 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:09.107566 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:03:09.904264 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:09.904226 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:03:10.106544 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:10.106508 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:03:10.106734 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:10.106584 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:03:11.106647 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:11.106608 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:03:11.107107 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:11.106675 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:03:12.107365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:12.107325 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:03:12.107768 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:12.107397 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:03:13.106465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:13.106431 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:03:13.106744 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:13.106508 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:03:14.107069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:14.107033 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:03:14.107599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:14.107090 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:03:15.110681 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:15.110648 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:15.111101 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:15.110927 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:15.111794 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:15.111777 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:03:23.174616 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:23.174581 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:03:25.363073 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.363035 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz"] Oct 30 22:03:25.373370 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.373336 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.421301 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.421260 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.421619 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.421595 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b698de93-8da9-45f6-954f-c8d7c64118d3-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.421865 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.421845 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.422034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.422016 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.422179 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.422164 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2ch\" (UniqueName: \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-kube-api-access-rr2ch\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.422277 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.422265 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.422381 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.422369 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.422526 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.422513 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.422680 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.422668 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.453178 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.453141 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz"] Oct 30 22:03:25.530665 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.530620 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.530850 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.530674 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b698de93-8da9-45f6-954f-c8d7c64118d3-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.530850 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.530743 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.530850 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.530787 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.530850 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.530825 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2ch\" (UniqueName: \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-kube-api-access-rr2ch\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.531128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.530851 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.531128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.530877 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.531128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.530932 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.531128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.531020 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.532108 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.531709 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.533970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.532330 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b698de93-8da9-45f6-954f-c8d7c64118d3-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.533970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.532565 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.533970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.532775 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.533970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.533243 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.536400 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.536360 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.537843 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.536800 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.624696 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.622372 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2ch\" (UniqueName: \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-kube-api-access-rr2ch\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.624696 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.624220 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:25.706484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:25.706166 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:26.198706 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:26.198677 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz"] Oct 30 22:03:26.199858 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:03:26.199827 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb698de93_8da9_45f6_954f_c8d7c64118d3.slice/crio-6e6e12ccf1bf98850eb28da57a0a77dc45ef85083fd8de992eadc1d6cd6c7858 WatchSource:0}: Error finding container 6e6e12ccf1bf98850eb28da57a0a77dc45ef85083fd8de992eadc1d6cd6c7858: Status 404 returned error can't find the container with id 6e6e12ccf1bf98850eb28da57a0a77dc45ef85083fd8de992eadc1d6cd6c7858 Oct 30 22:03:26.202448 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:26.202415 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:26.202540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:26.202495 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:26.202540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:26.202525 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:26.442816 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:26.442768 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" event={"ID":"b698de93-8da9-45f6-954f-c8d7c64118d3","Type":"ContainerStarted","Data":"60190fad34a88d979de72d2526ca01b451df973b02571109bd0ed987c69ce245"} Oct 30 22:03:26.443270 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:26.442838 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" event={"ID":"b698de93-8da9-45f6-954f-c8d7c64118d3","Type":"ContainerStarted","Data":"6e6e12ccf1bf98850eb28da57a0a77dc45ef85083fd8de992eadc1d6cd6c7858"} Oct 30 22:03:26.560983 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:26.560852 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" podStartSLOduration=1.5608327389999999 podStartE2EDuration="1.560832739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:26.558687283 +0000 UTC m=+1263.931314030" watchObservedRunningTime="2026-10-30 22:03:26.560832739 +0000 UTC m=+1263.933459486" Oct 30 22:03:26.709172 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:26.708341 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:27.708917 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:27.708884 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:03:27.709387 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:27.708940 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:03:27.879968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:27.877188 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7"] Oct 30 22:03:27.900382 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:27.900358 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:27.923938 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:27.923913 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7"] Oct 30 22:03:27.996282 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:27.996175 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgsxk\" (UniqueName: \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-kube-api-access-dgsxk\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:27.996458 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:27.996320 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-routing-overlay-volume\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:27.996458 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:27.996349 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-provider-credentials\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:27.996458 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:27.996433 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-plugins-config-volume\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:28.097338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.097299 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-plugins-config-volume\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:28.097535 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.097443 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgsxk\" (UniqueName: \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-kube-api-access-dgsxk\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:28.097535 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.097491 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-routing-overlay-volume\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:28.097535 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.097513 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-provider-credentials\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:28.098264 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.098234 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-plugins-config-volume\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:28.098398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.098238 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-routing-overlay-volume\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:28.100564 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.100543 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-provider-credentials\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:28.154449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.154421 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgsxk\" (UniqueName: \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-kube-api-access-dgsxk\") pod \"payload-processing-external-model-66f8fcd7fb-7jxn7\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:28.217755 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.217723 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:28.634251 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.634223 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7"] Oct 30 22:03:28.635052 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:03:28.635022 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5335d8bf_5b8d_495e_be60_8daf23381350.slice/crio-53c7535fc3cecd6db484f076612c98da3bec91d9d14880bbcb500b4cda1dbf34 WatchSource:0}: Error finding container 53c7535fc3cecd6db484f076612c98da3bec91d9d14880bbcb500b4cda1dbf34: Status 404 returned error can't find the container with id 53c7535fc3cecd6db484f076612c98da3bec91d9d14880bbcb500b4cda1dbf34 Oct 30 22:03:28.707418 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.707383 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:03:28.707588 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:28.707438 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:03:29.485974 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:29.484971 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" event={"ID":"5335d8bf-5b8d-495e-be60-8daf23381350","Type":"ContainerStarted","Data":"915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd"} Oct 30 22:03:29.485974 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:29.485042 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" event={"ID":"5335d8bf-5b8d-495e-be60-8daf23381350","Type":"ContainerStarted","Data":"53c7535fc3cecd6db484f076612c98da3bec91d9d14880bbcb500b4cda1dbf34"} Oct 30 22:03:29.485974 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:29.485934 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:29.602605 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:29.602544 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" podStartSLOduration=2.602529885 podStartE2EDuration="2.602529885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:29.598488768 +0000 UTC m=+1266.971115527" watchObservedRunningTime="2026-10-30 22:03:29.602529885 +0000 UTC m=+1266.975156630" Oct 30 22:03:29.707546 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:29.707509 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:03:29.707723 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:29.707561 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:03:30.707717 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:30.707682 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:03:30.708137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:30.707732 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:03:31.711527 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.711480 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:03:31.712098 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.711563 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:03:31.899366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.899332 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk"] Oct 30 22:03:31.903760 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.903742 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:31.990865 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.990661 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk"] Oct 30 22:03:31.997840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.997809 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:31.997983 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.997956 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:31.998034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.997988 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:31.998074 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.998037 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:31.998105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.998086 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3479d7ff-6792-481a-ad14-0672fb6ff04e-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:31.998159 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.998143 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:31.998211 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.998196 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:31.998246 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.998229 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:31.998278 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:31.998264 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z857\" (UniqueName: \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-kube-api-access-9z857\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.099608 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.099572 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.099783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.099646 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.099783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.099665 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.099783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.099699 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.099783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.099747 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3479d7ff-6792-481a-ad14-0672fb6ff04e-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.099922 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.099804 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.099922 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.099852 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.099922 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.099879 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.099922 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.099911 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z857\" (UniqueName: \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-kube-api-access-9z857\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.100143 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.100068 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.100326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.100301 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.100525 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.100496 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.100628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.100559 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.100920 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.100895 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3479d7ff-6792-481a-ad14-0672fb6ff04e-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.102128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.102106 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.102457 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.102439 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.156441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.156413 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z857\" (UniqueName: \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-kube-api-access-9z857\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.156584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.156567 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.220059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.220029 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:32.665596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.665566 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk"] Oct 30 22:03:32.667627 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:03:32.667589 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3479d7ff_6792_481a_ad14_0672fb6ff04e.slice/crio-c32667dc9cdec4ca5af34b7901b04711bdb6099ca3f2c30e53fcfc52bba8a0dd WatchSource:0}: Error finding container c32667dc9cdec4ca5af34b7901b04711bdb6099ca3f2c30e53fcfc52bba8a0dd: Status 404 returned error can't find the container with id c32667dc9cdec4ca5af34b7901b04711bdb6099ca3f2c30e53fcfc52bba8a0dd Oct 30 22:03:32.669868 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.669834 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:32.669980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.669900 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:32.669980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.669930 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:32.707639 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.707618 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:03:32.707749 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:32.707661 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:03:33.520212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:33.520169 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" event={"ID":"3479d7ff-6792-481a-ad14-0672fb6ff04e","Type":"ContainerStarted","Data":"f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e"} Oct 30 22:03:33.520609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:33.520234 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" event={"ID":"3479d7ff-6792-481a-ad14-0672fb6ff04e","Type":"ContainerStarted","Data":"c32667dc9cdec4ca5af34b7901b04711bdb6099ca3f2c30e53fcfc52bba8a0dd"} Oct 30 22:03:33.742022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:33.741962 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" podStartSLOduration=2.741928579 podStartE2EDuration="2.741928579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:33.690401151 +0000 UTC m=+1271.063027919" watchObservedRunningTime="2026-10-30 22:03:33.741928579 +0000 UTC m=+1271.114555326" Oct 30 22:03:33.812020 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:33.811914 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:33.812801 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:33.812780 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:33.813621 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:33.813598 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:03:34.220221 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:34.220186 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:34.221984 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:34.221959 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:03:34.222037 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:34.222017 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:03:35.220960 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:35.220914 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:03:35.221346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:35.221006 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:03:36.221340 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:36.221307 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:03:36.221706 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:36.221359 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:03:37.220806 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:37.220770 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:03:37.221015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:37.220823 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:03:37.802631 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:37.802597 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-32223ebd7979: secret "agc-cred-32223ebd7979" not found Oct 30 22:03:37.802631 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:37.802635 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6ede377aee21: secret "agc-cred-6ede377aee21" not found Oct 30 22:03:37.803094 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:37.802649 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-747d767c44-v9lrd: [secret "agc-cred-32223ebd7979" not found, secret "agc-cred-6ede377aee21" not found] Oct 30 22:03:37.803094 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:37.802709 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials podName:812d6776-ca57-42f7-afe1-27f4fcc03cae nodeName:}" failed. No retries permitted until 2026-10-30 22:03:38.30268829 +0000 UTC m=+1275.675315037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials") pod "payload-processing-external-model-747d767c44-v9lrd" (UID: "812d6776-ca57-42f7-afe1-27f4fcc03cae") : [secret "agc-cred-32223ebd7979" not found, secret "agc-cred-6ede377aee21" not found] Oct 30 22:03:38.220507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:38.220475 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:03:38.220669 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:38.220533 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:03:38.310646 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:38.310607 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-32223ebd7979: secret "agc-cred-32223ebd7979" not found Oct 30 22:03:38.310646 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:38.310648 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6ede377aee21: secret "agc-cred-6ede377aee21" not found Oct 30 22:03:38.310646 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:38.310657 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-747d767c44-v9lrd: [secret "agc-cred-32223ebd7979" not found, secret "agc-cred-6ede377aee21" not found] Oct 30 22:03:38.310884 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:38.310706 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials podName:812d6776-ca57-42f7-afe1-27f4fcc03cae nodeName:}" failed. No retries permitted until 2026-10-30 22:03:39.310691972 +0000 UTC m=+1276.683318696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials") pod "payload-processing-external-model-747d767c44-v9lrd" (UID: "812d6776-ca57-42f7-afe1-27f4fcc03cae") : [secret "agc-cred-32223ebd7979" not found, secret "agc-cred-6ede377aee21" not found] Oct 30 22:03:39.220451 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:39.220417 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:03:39.220846 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:39.220473 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:03:39.319794 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:39.319764 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-32223ebd7979: secret "agc-cred-32223ebd7979" not found Oct 30 22:03:39.319794 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:39.319793 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6ede377aee21: secret "agc-cred-6ede377aee21" not found Oct 30 22:03:39.320004 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:39.319801 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-747d767c44-v9lrd: [secret "agc-cred-32223ebd7979" not found, secret "agc-cred-6ede377aee21" not found] Oct 30 22:03:39.320004 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:39.319851 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials podName:812d6776-ca57-42f7-afe1-27f4fcc03cae nodeName:}" failed. No retries permitted until 2026-10-30 22:03:41.319835795 +0000 UTC m=+1278.692462536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials") pod "payload-processing-external-model-747d767c44-v9lrd" (UID: "812d6776-ca57-42f7-afe1-27f4fcc03cae") : [secret "agc-cred-32223ebd7979" not found, secret "agc-cred-6ede377aee21" not found] Oct 30 22:03:39.879137 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:39.879094 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:03:40.225335 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:40.225247 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:40.225785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:40.225663 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:40.226492 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:40.226474 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:03:40.545968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:40.545866 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7"] Oct 30 22:03:40.548677 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:40.548639 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" podUID="5335d8bf-5b8d-495e-be60-8daf23381350" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:03:40.549363 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:40.549345 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-747d767c44-v9lrd"] Oct 30 22:03:41.349237 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:41.349198 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-32223ebd7979: secret "agc-cred-32223ebd7979" not found Oct 30 22:03:41.349792 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:41.349771 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-501bd36994fb: secret "agc-cred-501bd36994fb" not found Oct 30 22:03:41.349898 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:41.349888 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6ede377aee21: secret "agc-cred-6ede377aee21" not found Oct 30 22:03:41.350014 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:41.350002 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-747d767c44-v9lrd: [secret "agc-cred-32223ebd7979" not found, secret "agc-cred-501bd36994fb" not found, secret "agc-cred-6ede377aee21" not found] Oct 30 22:03:41.352116 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:41.352091 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials podName:812d6776-ca57-42f7-afe1-27f4fcc03cae nodeName:}" failed. No retries permitted until 2026-10-30 22:03:45.350142688 +0000 UTC m=+1282.722769416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials") pod "payload-processing-external-model-747d767c44-v9lrd" (UID: "812d6776-ca57-42f7-afe1-27f4fcc03cae") : [secret "agc-cred-32223ebd7979" not found, secret "agc-cred-501bd36994fb" not found, secret "agc-cred-6ede377aee21" not found] Oct 30 22:03:45.398540 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:45.398508 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-32223ebd7979: secret "agc-cred-32223ebd7979" not found Oct 30 22:03:45.398540 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:45.398537 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-501bd36994fb: secret "agc-cred-501bd36994fb" not found Oct 30 22:03:45.398540 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:45.398544 4859 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6ede377aee21: secret "agc-cred-6ede377aee21" not found Oct 30 22:03:45.398997 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:45.398553 4859 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-747d767c44-v9lrd: [secret "agc-cred-32223ebd7979" not found, secret "agc-cred-501bd36994fb" not found, secret "agc-cred-6ede377aee21" not found] Oct 30 22:03:45.398997 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:45.398607 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials podName:812d6776-ca57-42f7-afe1-27f4fcc03cae nodeName:}" failed. No retries permitted until 2026-10-30 22:03:53.398593469 +0000 UTC m=+1290.771220209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials") pod "payload-processing-external-model-747d767c44-v9lrd" (UID: "812d6776-ca57-42f7-afe1-27f4fcc03cae") : [secret "agc-cred-32223ebd7979" not found, secret "agc-cred-501bd36994fb" not found, secret "agc-cred-6ede377aee21" not found] Oct 30 22:03:45.546957 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.546910 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" podUID="5335d8bf-5b8d-495e-be60-8daf23381350" containerName="payload-processing" containerID="cri-o://915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd" gracePeriod=25 Oct 30 22:03:45.551596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.551556 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" podUID="812d6776-ca57-42f7-afe1-27f4fcc03cae" containerName="payload-processing" containerID="cri-o://2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934" gracePeriod=25 Oct 30 22:03:45.822962 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.822922 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:45.851145 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.851121 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:03:45.905137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.905112 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-routing-overlay-volume\") pod \"812d6776-ca57-42f7-afe1-27f4fcc03cae\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " Oct 30 22:03:45.905271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.905258 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-provider-credentials\") pod \"5335d8bf-5b8d-495e-be60-8daf23381350\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " Oct 30 22:03:45.905326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.905285 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials\") pod \"812d6776-ca57-42f7-afe1-27f4fcc03cae\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " Oct 30 22:03:45.905326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.905316 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-plugins-config-volume\") pod \"5335d8bf-5b8d-495e-be60-8daf23381350\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " Oct 30 22:03:45.905432 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.905338 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-kube-api-access-dgsxk\" (UniqueName: \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-kube-api-access-dgsxk\") pod \"5335d8bf-5b8d-495e-be60-8daf23381350\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " Oct 30 22:03:45.905485 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.905463 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-kube-api-access-trmhw\" (UniqueName: \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-kube-api-access-trmhw\") pod \"812d6776-ca57-42f7-afe1-27f4fcc03cae\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " Oct 30 22:03:45.905537 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.905510 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-plugins-config-volume\") pod \"812d6776-ca57-42f7-afe1-27f4fcc03cae\" (UID: \"812d6776-ca57-42f7-afe1-27f4fcc03cae\") " Oct 30 22:03:45.905604 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.905585 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-routing-overlay-volume\") pod \"5335d8bf-5b8d-495e-be60-8daf23381350\" (UID: \"5335d8bf-5b8d-495e-be60-8daf23381350\") " Oct 30 22:03:45.905659 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.905466 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-routing-overlay-volume" pod "812d6776-ca57-42f7-afe1-27f4fcc03cae" (UID: "812d6776-ca57-42f7-afe1-27f4fcc03cae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:45.905738 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.905716 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-plugins-config-volume" pod "5335d8bf-5b8d-495e-be60-8daf23381350" (UID: "5335d8bf-5b8d-495e-be60-8daf23381350"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:45.906305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.906128 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-routing-overlay-volume" pod "5335d8bf-5b8d-495e-be60-8daf23381350" (UID: "5335d8bf-5b8d-495e-be60-8daf23381350"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:45.906305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.906241 4859 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-routing-overlay-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:45.906305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.906242 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-plugins-config-volume" pod "812d6776-ca57-42f7-afe1-27f4fcc03cae" (UID: "812d6776-ca57-42f7-afe1-27f4fcc03cae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:45.906305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.906260 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:45.906305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.906278 4859 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5335d8bf-5b8d-495e-be60-8daf23381350-routing-overlay-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:45.907729 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.907704 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-provider-credentials" pod "5335d8bf-5b8d-495e-be60-8daf23381350" (UID: "5335d8bf-5b8d-495e-be60-8daf23381350"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:45.907876 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.907854 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials" pod "812d6776-ca57-42f7-afe1-27f4fcc03cae" (UID: "812d6776-ca57-42f7-afe1-27f4fcc03cae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:45.908039 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.907932 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-kube-api-access-dgsxk" pod "5335d8bf-5b8d-495e-be60-8daf23381350" (UID: "5335d8bf-5b8d-495e-be60-8daf23381350"). InnerVolumeSpecName "kube-api-access-dgsxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:45.908231 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:45.908211 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-kube-api-access-trmhw" pod "812d6776-ca57-42f7-afe1-27f4fcc03cae" (UID: "812d6776-ca57-42f7-afe1-27f4fcc03cae"). InnerVolumeSpecName "kube-api-access-trmhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:46.008170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.008096 4859 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-provider-credentials\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.008170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.008132 4859 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-provider-credentials\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.008170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.008147 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgsxk\" (UniqueName: \"kubernetes.io/projected/5335d8bf-5b8d-495e-be60-8daf23381350-kube-api-access-dgsxk\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.008170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.008162 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trmhw\" (UniqueName: \"kubernetes.io/projected/812d6776-ca57-42f7-afe1-27f4fcc03cae-kube-api-access-trmhw\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.008398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.008176 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/812d6776-ca57-42f7-afe1-27f4fcc03cae-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.688966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.688570 4859 generic.go:356] "Generic (PLEG): container finished" podID="5335d8bf-5b8d-495e-be60-8daf23381350" containerID="915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd" exitCode=0 Oct 30 22:03:46.688966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.688788 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" event={"ID":"5335d8bf-5b8d-495e-be60-8daf23381350","Type":"ContainerDied","Data":"915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd"} Oct 30 22:03:46.688966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.688843 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" event={"ID":"5335d8bf-5b8d-495e-be60-8daf23381350","Type":"ContainerDied","Data":"53c7535fc3cecd6db484f076612c98da3bec91d9d14880bbcb500b4cda1dbf34"} Oct 30 22:03:46.688966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.688872 4859 scope.go:122] "RemoveContainer" containerID="915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd" Oct 30 22:03:46.689632 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.689061 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7" Oct 30 22:03:46.728980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.709760 4859 generic.go:356] "Generic (PLEG): container finished" podID="812d6776-ca57-42f7-afe1-27f4fcc03cae" containerID="2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934" exitCode=0 Oct 30 22:03:46.728980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.709814 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" event={"ID":"812d6776-ca57-42f7-afe1-27f4fcc03cae","Type":"ContainerDied","Data":"2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934"} Oct 30 22:03:46.728980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.709863 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" event={"ID":"812d6776-ca57-42f7-afe1-27f4fcc03cae","Type":"ContainerDied","Data":"7ca0d3b68355628840b923c4a25264d91c61b0a71cce500f134eb59901737389"} Oct 30 22:03:46.728980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.709957 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747d767c44-v9lrd" Oct 30 22:03:46.985185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.985147 4859 scope.go:122] "RemoveContainer" containerID="915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd" Oct 30 22:03:46.986020 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:46.985855 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd\": container with ID starting with 915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd not found: ID does not exist" containerID="915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd" Oct 30 22:03:46.986020 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.985897 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd"} err="failed to get container status \"915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd\": rpc error: code = NotFound desc = could not find container \"915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd\": container with ID starting with 915832dc0f132efc667f74b96caf292ee05aea2eedd8ad32f249395a5139febd not found: ID does not exist" Oct 30 22:03:46.986020 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:46.985922 4859 scope.go:122] "RemoveContainer" containerID="2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934" Oct 30 22:03:47.072186 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:47.072150 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-747d767c44-v9lrd"] Oct 30 22:03:47.107314 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:47.107228 4859 scope.go:122] "RemoveContainer" containerID="2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934" Oct 30 22:03:47.108160 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:03:47.108115 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934\": container with ID starting with 2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934 not found: ID does not exist" containerID="2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934" Oct 30 22:03:47.108370 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:47.108339 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934"} err="failed to get container status \"2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934\": rpc error: code = NotFound desc = could not find container \"2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934\": container with ID starting with 2c0f4cf4aa76cc3c817ad0026ea614183ee04315ee88ca8a3e0de08b026cf934 not found: ID does not exist" Oct 30 22:03:47.110183 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:47.110162 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-747d767c44-v9lrd"] Oct 30 22:03:47.221034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:47.220996 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7"] Oct 30 22:03:47.237609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:47.237578 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f8fcd7fb-7jxn7"] Oct 30 22:03:48.758187 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:48.758145 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:48.918969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:48.914778 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 22:03:48.954558 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:48.954516 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:49.078328 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:49.009199 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5335d8bf-5b8d-495e-be60-8daf23381350" path="/var/lib/kubelet/pods/5335d8bf-5b8d-495e-be60-8daf23381350/volumes" Oct 30 22:03:49.085441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:49.078666 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="812d6776-ca57-42f7-afe1-27f4fcc03cae" path="/var/lib/kubelet/pods/812d6776-ca57-42f7-afe1-27f4fcc03cae/volumes" Oct 30 22:03:49.154491 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:49.154441 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:49.161992 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:49.158030 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:49.768186 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:49.768139 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:49.769346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:49.769320 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 22:03:49.932690 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:49.928534 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:49.932690 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:49.929333 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:03:49.987474 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:49.986174 4859 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:50.040615 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:50.038741 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:50.040615 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:50.038820 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:50.056983 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:50.056072 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mgw8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:50.056983 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:50.056153 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" podUID="b6e11781-e771-4f4d-8458-3966ee881c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:50.440170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:50.440108 4859 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:50.916286 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:50.916227 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:50.932377 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:50.932334 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:51.118968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:51.107234 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 22:03:51.145868 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:51.145836 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:03:51.195397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:51.195007 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6sd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:51.195397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:51.195068 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:51.195701 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:51.195676 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:51.195829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:51.195807 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:52.426358 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.426327 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p"] Oct 30 22:03:52.426881 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.426867 4859 cpu_manager.go:414] "containerMap: removing container" podUID="5335d8bf-5b8d-495e-be60-8daf23381350" containerName="payload-processing" Oct 30 22:03:52.426926 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.426881 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5335d8bf-5b8d-495e-be60-8daf23381350" containerName="payload-processing" Oct 30 22:03:52.426926 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.426907 4859 cpu_manager.go:414] "containerMap: removing container" podUID="812d6776-ca57-42f7-afe1-27f4fcc03cae" containerName="payload-processing" Oct 30 22:03:52.426926 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.426912 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="812d6776-ca57-42f7-afe1-27f4fcc03cae" containerName="payload-processing" Oct 30 22:03:52.427069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.427007 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5335d8bf-5b8d-495e-be60-8daf23381350" containerName="payload-processing" Oct 30 22:03:52.427069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.427024 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="812d6776-ca57-42f7-afe1-27f4fcc03cae" containerName="payload-processing" Oct 30 22:03:52.432059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.432044 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.512539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.512486 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.512728 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.512596 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.512728 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.512641 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vll9j\" (UniqueName: \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-kube-api-access-vll9j\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.512869 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.512740 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.512869 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.512792 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.512869 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.512851 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.513051 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.512909 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.513106 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.513067 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.513185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.513152 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.538295 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.538268 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p"] Oct 30 22:03:52.614416 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.614385 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.614604 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.614424 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.614604 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.614443 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.614604 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.614503 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.614604 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.614555 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.614814 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.614629 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.614814 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.614676 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.614814 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.614699 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vll9j\" (UniqueName: \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-kube-api-access-vll9j\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.614974 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.614920 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.615044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.614968 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.615110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.615072 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.615168 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.615101 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.615306 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.615284 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.615385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.615312 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.616734 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.616708 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.617283 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.617265 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.670994 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.670965 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.671122 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.671099 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vll9j\" (UniqueName: \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-kube-api-access-vll9j\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:52.746139 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:52.746060 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:53.188907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:53.188880 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p"] Oct 30 22:03:53.194896 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:53.194852 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:53.195003 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:53.194958 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:53.195075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:53.195057 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:53.303224 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:53.303168 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" event={"ID":"3ed1f318-6c24-4904-8a3d-64a71fcab80e","Type":"ContainerStarted","Data":"02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b"} Oct 30 22:03:53.303538 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:53.303485 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" event={"ID":"3ed1f318-6c24-4904-8a3d-64a71fcab80e","Type":"ContainerStarted","Data":"34d79e68b62a3064c04c543c66443a1d020d0ecd519b564196926ae3d17b2b56"} Oct 30 22:03:53.423854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:53.423789 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" podStartSLOduration=1.4237693089999999 podStartE2EDuration="1.423769309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:53.421278081 +0000 UTC m=+1290.793904830" watchObservedRunningTime="2026-10-30 22:03:53.423769309 +0000 UTC m=+1290.796396056" Oct 30 22:03:53.747187 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:53.747157 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:54.748097 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:54.748062 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:03:54.748471 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:54.748109 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:03:55.747179 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:55.747145 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:03:55.747366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:55.747194 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:03:56.746595 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:56.746560 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:03:56.747073 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:56.746618 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:03:57.747427 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:57.747394 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:03:57.747798 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:57.747445 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:03:58.747027 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:58.746996 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:03:58.747197 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:58.747055 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:03:59.944676 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:59.944636 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:59.945327 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:59.945302 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:03:59.946096 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:03:59.946074 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:04:02.162798 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.162763 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b"] Oct 30 22:04:02.168064 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.168038 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.248033 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.247999 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b"] Oct 30 22:04:02.260209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.260172 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.260406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.260229 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.260406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.260293 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.260406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.260348 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfn7m\" (UniqueName: \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-kube-api-access-vfn7m\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.260406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.260377 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.260626 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.260426 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.260626 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.260467 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.260626 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.260495 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.260626 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.260518 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.361570 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.361532 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.361785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.361645 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.361785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.361684 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.361785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.361708 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.361785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.361770 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfn7m\" (UniqueName: \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-kube-api-access-vfn7m\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.362152 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.362128 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.362248 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.362228 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.362332 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.362317 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.362412 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.362394 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.362627 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.362598 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.362627 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.362493 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.362798 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.362561 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.362798 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.362644 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.362932 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.362911 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.364741 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.364718 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.364831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.364754 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.429370 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.429289 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfn7m\" (UniqueName: \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-kube-api-access-vfn7m\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.429370 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.429298 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.482813 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.482775 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:02.911588 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.911554 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b"] Oct 30 22:04:02.913916 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:04:02.913870 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef6c3cb_d4e0_4770_8a85_919938bd155a.slice/crio-5f2b552a73037659099575120fd10cc0b6363c363508ba4b6002be37c0f2b0f0 WatchSource:0}: Error finding container 5f2b552a73037659099575120fd10cc0b6363c363508ba4b6002be37c0f2b0f0: Status 404 returned error can't find the container with id 5f2b552a73037659099575120fd10cc0b6363c363508ba4b6002be37c0f2b0f0 Oct 30 22:04:02.918969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.918921 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:02.919068 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.919012 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:02.919068 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:02.919044 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:03.121513 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.121472 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565"] Oct 30 22:04:03.127000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.126971 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.176615 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.176514 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-model-cache\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.177065 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.176625 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-dshm\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.177065 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.176742 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-tmp-dir\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.177065 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.176807 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qg28\" (UniqueName: \"kubernetes.io/projected/ec42687d-7e39-4655-8873-8f790f18baf6-kube-api-access-6qg28\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.177065 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.176852 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec42687d-7e39-4655-8873-8f790f18baf6-tls-certs\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.177065 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.176904 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-home\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.180292 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.180254 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"openshift-service-ca.crt\"" Oct 30 22:04:03.180530 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.180512 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"default-dockercfg-6s4ht\"" Oct 30 22:04:03.180693 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.180679 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"kube-root-ca.crt\"" Oct 30 22:04:03.181447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.181426 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"distinct-2-w0-b8717e-kserve-self-signed-certs\"" Oct 30 22:04:03.186858 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.186833 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565"] Oct 30 22:04:03.287268 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.287238 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-model-cache\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.287527 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.287506 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-dshm\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.287708 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.287689 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-tmp-dir\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.287896 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.287877 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qg28\" (UniqueName: \"kubernetes.io/projected/ec42687d-7e39-4655-8873-8f790f18baf6-kube-api-access-6qg28\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.288043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.288026 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec42687d-7e39-4655-8873-8f790f18baf6-tls-certs\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.288170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.288156 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-home\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.288784 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.288762 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-home\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.289196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.289176 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-model-cache\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.291035 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.291014 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-tmp-dir\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.308641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.308610 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec42687d-7e39-4655-8873-8f790f18baf6-tls-certs\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.309300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.309176 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-dshm\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.349144 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.349083 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qg28\" (UniqueName: \"kubernetes.io/projected/ec42687d-7e39-4655-8873-8f790f18baf6-kube-api-access-6qg28\") pod \"distinct-2-w0-b8717e-kserve-866884dffb-8j565\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.381779 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.381741 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" event={"ID":"8ef6c3cb-d4e0-4770-8a85-919938bd155a","Type":"ContainerStarted","Data":"d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf"} Oct 30 22:04:03.381976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.381828 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" event={"ID":"8ef6c3cb-d4e0-4770-8a85-919938bd155a","Type":"ContainerStarted","Data":"5f2b552a73037659099575120fd10cc0b6363c363508ba4b6002be37c0f2b0f0"} Oct 30 22:04:03.445533 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.445446 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:03.541162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.541085 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" podStartSLOduration=1.5410632290000001 podStartE2EDuration="1.541063229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:03.50842003 +0000 UTC m=+1300.881046776" watchObservedRunningTime="2026-10-30 22:04:03.541063229 +0000 UTC m=+1300.913689975" Oct 30 22:04:03.572981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.571928 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:03.598004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.579330 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:04:03.598004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.579404 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:04:03.969680 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:03.969513 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565"] Oct 30 22:04:04.399338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:04.399303 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" event={"ID":"ec42687d-7e39-4655-8873-8f790f18baf6","Type":"ContainerStarted","Data":"7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea"} Oct 30 22:04:04.399338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:04.399346 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" event={"ID":"ec42687d-7e39-4655-8873-8f790f18baf6","Type":"ContainerStarted","Data":"c3afd91673672adb10e6ea568319692d923e4698d3029949a358ada5b2fe93aa"} Oct 30 22:04:04.445611 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:04.445584 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:04.490470 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:04.490428 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:04:04.490660 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:04.490496 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:04:04.517375 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:04.517318 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" podStartSLOduration=1.517304503 podStartE2EDuration="1.517304503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:04.513208562 +0000 UTC m=+1301.885835307" watchObservedRunningTime="2026-10-30 22:04:04.517304503 +0000 UTC m=+1301.889931248" Oct 30 22:04:05.485903 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:05.485373 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:04:05.485903 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:05.485438 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:04:06.485968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:06.485400 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:04:06.485968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:06.485463 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:04:07.248016 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.247981 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p"] Oct 30 22:04:07.252165 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.252148 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.298493 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.298459 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt"] Oct 30 22:04:07.302564 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.302540 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.347923 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.347890 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p"] Oct 30 22:04:07.401001 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.394157 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.401308 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.401277 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.401467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.401447 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p88s\" (UniqueName: \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-kube-api-access-8p88s\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.401683 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.401607 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.401683 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.401659 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.401846 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.401700 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.401846 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.401750 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.401846 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.401797 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9293cc9-9ba6-4328-a72d-df484d2388e1-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.401846 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.401835 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2h9\" (UniqueName: \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-kube-api-access-hh2h9\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.402090 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.401893 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.402090 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.401964 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.402090 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.402004 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.402243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.402087 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.402243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.402127 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.402243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.402169 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.402243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.402193 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.402243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.402217 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.402475 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.402274 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.412778 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.412753 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt"] Oct 30 22:04:07.484439 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.484402 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:04:07.492094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.484688 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:04:07.503792 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.503708 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.503792 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.503770 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504035 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.503814 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9293cc9-9ba6-4328-a72d-df484d2388e1-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504121 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504098 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504281 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504259 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2h9\" (UniqueName: \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-kube-api-access-hh2h9\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.504357 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504303 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504357 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504346 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.504464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504405 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.504464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504447 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504567 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504511 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.504567 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504551 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.504666 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504595 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9293cc9-9ba6-4328-a72d-df484d2388e1-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504666 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504610 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504666 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504647 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504811 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504675 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.504811 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504726 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504811 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504739 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.504997 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504823 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504997 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504873 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.504997 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504899 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p88s\" (UniqueName: \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-kube-api-access-8p88s\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.504997 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.504972 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.505200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.505008 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.505200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.505116 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.505307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.505259 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.505364 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.505328 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.505419 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.505385 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.505571 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.505550 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.505784 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.505766 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.507045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.507003 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.509883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.509089 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.509883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.509393 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.510589 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.510565 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.576999 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.561531 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2h9\" (UniqueName: \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-kube-api-access-hh2h9\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.576999 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.575217 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p88s\" (UniqueName: \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-kube-api-access-8p88s\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.585982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.584764 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.601678 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.600778 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:07.616992 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.616962 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:07.879449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:07.879181 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:08.101175 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.101141 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt"] Oct 30 22:04:08.110730 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.108921 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:08.110730 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.109046 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:08.110730 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.109090 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:08.306923 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.305894 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:08.306923 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.306016 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:08.306923 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.306068 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:08.307268 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.307174 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p"] Oct 30 22:04:08.439702 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.438602 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" event={"ID":"e9293cc9-9ba6-4328-a72d-df484d2388e1","Type":"ContainerStarted","Data":"6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3"} Oct 30 22:04:08.439702 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.438672 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" event={"ID":"e9293cc9-9ba6-4328-a72d-df484d2388e1","Type":"ContainerStarted","Data":"72c0bb5124604447aae2843af74df6db0e28d2f325f014ce9d19e1e164152c2d"} Oct 30 22:04:08.441628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.441600 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" event={"ID":"0630b556-6459-480b-abf9-a3f3d0ecc6c1","Type":"ContainerStarted","Data":"8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a"} Oct 30 22:04:08.441752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.441657 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" event={"ID":"0630b556-6459-480b-abf9-a3f3d0ecc6c1","Type":"ContainerStarted","Data":"d7801f62f4c8c4a988773267985e7f45d050263eda90b32822a28c574248109b"} Oct 30 22:04:08.483966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.483355 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:04:08.483966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.483413 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:04:08.557359 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.557252 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podStartSLOduration=1.557233866 podStartE2EDuration="1.557233866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:08.55656421 +0000 UTC m=+1305.929190958" watchObservedRunningTime="2026-10-30 22:04:08.557233866 +0000 UTC m=+1305.929860615" Oct 30 22:04:08.617882 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.617849 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:08.684202 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.684082 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podStartSLOduration=1.684059953 podStartE2EDuration="1.684059953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:08.678423667 +0000 UTC m=+1306.051050437" watchObservedRunningTime="2026-10-30 22:04:08.684059953 +0000 UTC m=+1306.056686700" Oct 30 22:04:08.879796 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:08.879754 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:09.485976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:09.485470 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:04:09.485976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:09.485538 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:04:09.622969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:09.621055 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:09.622969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:09.621118 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:09.888969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:09.888235 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:09.888969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:09.888309 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:09.957254 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:04:09.956931 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:04:10.554157 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.554116 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8"] Oct 30 22:04:10.559247 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.559220 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:10.607972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.588490 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:04:10.631018 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.624373 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:10.631018 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.624434 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:10.772969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.761526 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:10.845374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.845335 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdk6t\" (UniqueName: \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-kube-api-access-zdk6t\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:10.855395 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.855266 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:10.872500 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.855621 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:10.914001 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.887145 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:10.914001 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.887225 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:10.914001 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.887293 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:10.914001 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.887347 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:10.914001 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.887395 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:10.929014 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.918323 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:10.929399 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.929285 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:10.951472 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.949529 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:10.951472 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.949593 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8"] Oct 30 22:04:10.951472 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.951158 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:10.953370 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:10.953346 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:04:11.008032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.006819 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.006882 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.006999 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.007026 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.007071 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.007115 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.007149 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.007228 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.007259 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdk6t\" (UniqueName: \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-kube-api-access-zdk6t\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.008410 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008718 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.008667 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.008990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.008964 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.015973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.009907 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.015973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.015875 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.020074 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.018534 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.020882 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.020772 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.107928 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.107854 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdk6t\" (UniqueName: \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-kube-api-access-zdk6t\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.125158 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.124315 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.226970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.226296 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:11.628524 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.628486 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:11.628778 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.628746 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:11.965707 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.965670 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:11.994718 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:11.994669 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:12.280803 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.263295 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8"] Oct 30 22:04:12.290976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.290502 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:12.290976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.290616 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:12.290976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.290661 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:12.618657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.618220 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:12.618657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.618282 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:12.706249 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.706169 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" event={"ID":"784a3a6f-5419-48d6-a7da-3eb93c205b9b","Type":"ContainerStarted","Data":"503283aaff860097a6ee354faf5f665c510d86bf31768b2221206527ae7237d9"} Oct 30 22:04:12.884790 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.884663 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:12.884790 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.884724 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:12.930848 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.930038 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.988965 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:12.988249 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:04:13.155106 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:13.155056 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 22:04:13.449904 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:13.447683 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:13.626969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:13.619361 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:13.626969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:13.619443 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:13.822033 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:13.819689 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" event={"ID":"784a3a6f-5419-48d6-a7da-3eb93c205b9b","Type":"ContainerStarted","Data":"07aa472f1aa2bea104b6a09c5bf01d45886ed86ec98a64f9de77a6d0b2565a26"} Oct 30 22:04:13.894968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:13.882836 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:13.894968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:13.882896 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:13.948995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:13.932583 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:04:13.948995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:13.932908 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 22:04:14.093599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.046044 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:14.093599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.046469 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 22:04:14.161978 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.161062 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 22:04:14.161978 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.161576 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:04:14.241990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.232191 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:14.324563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.314544 4859 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:14.409025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.406590 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:14.409025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.406652 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:14.413979 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.409689 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podStartSLOduration=4.409671032 podStartE2EDuration="4.409671032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:14.015613576 +0000 UTC m=+1311.388240323" watchObservedRunningTime="2026-10-30 22:04:14.409671032 +0000 UTC m=+1311.782297779" Oct 30 22:04:14.483969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.456087 4859 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" podUID="ec42687d-7e39-4655-8873-8f790f18baf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:14.672053 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.644028 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:14.674077 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.674028 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:14.910031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.902321 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:14.925136 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.925027 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:14.943545 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:14.943494 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.081996 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:15.080810 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.351753 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:15.351711 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:15.352373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:15.352336 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:15.352916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:15.352884 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.644059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:15.633677 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:15.644059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:15.633749 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:16.060025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.054937 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.064382 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:04:16.064347 4859 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.321s" Oct 30 22:04:16.082968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.081076 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.107969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.097437 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:16.107969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.097514 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.219510 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.219470 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:16.228339 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.228291 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.364045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.363518 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:16.364045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.363579 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:16.415996 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.385914 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.462969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.462828 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:16.462969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.462913 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:16.646165 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.646107 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:16.646507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.646453 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:16.893728 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.893631 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:16.894012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:16.893980 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:17.023307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.023275 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:04:17.078315 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.078275 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:04:17.092094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.092042 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:17.168106 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.147067 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:17.168106 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.147162 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:17.314101 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.274487 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:17.314101 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.274550 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:17.635685 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.635538 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:17.635685 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.635602 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:17.923029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.898397 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:17.923029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:17.898458 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:18.107178 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:18.081913 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:18.110893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:18.109183 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:18.263028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:18.262072 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:18.263028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:18.262134 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:18.463652 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:18.373981 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:18.678162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:18.675526 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:18.678162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:18.675586 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:18.898289 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:18.898133 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:18.898289 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:18.898207 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:19.172327 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.172277 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:19.247174 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.181926 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:19.247174 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.245545 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:19.247174 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.245626 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:19.388269 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.388219 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:19.640728 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.640690 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:04:19.655541 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.655502 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:19.658110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.658070 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:19.889158 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.889106 4859 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:19.892246 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.892117 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:19.892823 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:19.892600 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:20.036632 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.036583 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 22:04:20.156202 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.145374 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 22:04:20.160992 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.160746 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:04:20.222025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.197874 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h"] Oct 30 22:04:20.232622 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.227406 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.233220 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.233094 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:20.233220 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.233144 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:20.236463 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.234487 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h"] Oct 30 22:04:20.297036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.297004 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7fabe91d-openshift-default-dockercfg-rt9jz\"" Oct 30 22:04:20.331272 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.330924 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.331450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.331314 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.331521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.331449 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.331521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.331503 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.331634 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.331549 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6373d52e-4048-41f8-a998-1532d9ed4403-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.331634 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.331590 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6373d52e-4048-41f8-a998-1532d9ed4403-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.331728 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.331632 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.331779 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.331762 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.527545 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.527431 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhgx4\" (UniqueName: \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-kube-api-access-zhgx4\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.527809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.527785 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.527966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.527932 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.528229 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.528210 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.528398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.528380 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.528582 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.528549 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6373d52e-4048-41f8-a998-1532d9ed4403-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.528745 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.528729 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6373d52e-4048-41f8-a998-1532d9ed4403-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.528872 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.528858 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.529085 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.529068 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.529866 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.529846 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.530331 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.530312 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.531477 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.531456 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.531897 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.531855 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.532762 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.532744 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6373d52e-4048-41f8-a998-1532d9ed4403-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.621503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.621440 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:20.622179 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.622108 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:20.636409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.636359 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.668439 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.668383 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhgx4\" (UniqueName: \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-kube-api-access-zhgx4\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.701964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.701912 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6373d52e-4048-41f8-a998-1532d9ed4403-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.843246 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.748807 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.918809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.918771 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:04:20.934226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.933505 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:04:20.934226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.934106 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhgx4\" (UniqueName: \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-kube-api-access-zhgx4\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:20.954669 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:20.953735 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:21.237373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:21.237244 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:21.237373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:21.237304 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:21.616547 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:21.613507 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h"] Oct 30 22:04:21.631490 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:21.629543 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:21.631490 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:21.629635 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:21.631490 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:21.629677 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:21.634511 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:21.617309 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:04:21.634511 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:21.633102 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:04:22.232693 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.232656 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:22.233324 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.233287 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:22.392971 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.388222 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:22.396973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.393874 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:22.397747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.397717 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" event={"ID":"6373d52e-4048-41f8-a998-1532d9ed4403","Type":"ContainerStarted","Data":"a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269"} Oct 30 22:04:22.398686 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.398664 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" event={"ID":"6373d52e-4048-41f8-a998-1532d9ed4403","Type":"ContainerStarted","Data":"c50fe97698f33645b4142f1be8e2caabb649b59319ddd496c9cfe4862be3b50c"} Oct 30 22:04:22.406583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.405230 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:04:22.617104 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.617068 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:22.645163 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.645103 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" podStartSLOduration=2.645088708 podStartE2EDuration="2.645088708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:22.6438708 +0000 UTC m=+1320.016497545" watchObservedRunningTime="2026-10-30 22:04:22.645088708 +0000 UTC m=+1320.017715453" Oct 30 22:04:22.954814 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.954726 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:22.960236 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:22.960120 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:23.228427 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:23.228342 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:23.228596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:23.228420 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:23.449205 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:23.449173 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:23.451965 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:23.451912 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:04:23.530022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:23.529240 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:23.548833 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:23.548792 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:04:23.660746 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:23.660689 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:23.660922 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:23.660779 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:24.228496 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:24.228460 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:04:24.228696 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:24.228527 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 22:04:24.291123 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:24.291071 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:24.292722 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:24.292694 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:04:25.285549 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:25.285513 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:25.286015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:25.285990 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:25.286829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:25.286808 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:04:33.339845 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:33.339811 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g"] Oct 30 22:04:33.340359 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:33.340092 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" containerID="cri-o://928a09032ef8dac95cf573e67859bfad56e0377885d8a1ec41b4be70c5a5ba3b" gracePeriod=30 Oct 30 22:04:35.023965 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:35.023595 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:04:35.023965 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:35.023680 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:04:36.960970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:36.960794 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:37.273055 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:37.252164 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:04:37.273055 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:37.252599 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 22:04:37.787972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:37.787233 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:37.787972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:37.787318 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:38.031967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:38.024423 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:38.031967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:38.026008 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 22:04:38.299969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:38.299016 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:04:38.309973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:38.300537 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 22:04:38.332070 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:38.332023 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 22:04:38.333640 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:38.333614 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:04:38.479468 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:38.479422 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:38.479758 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:38.479726 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:38.718969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:38.717092 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" podUID="ec42687d-7e39-4655-8873-8f790f18baf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.070072 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.064724 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.321019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.315025 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" podUID="27da8df3-0147-4496-9997-2e55b71fa988" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:04:39.340968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.330768 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:04:39.340968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.338314 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 22:04:39.340968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.338876 4859 generic.go:356] "Generic (PLEG): container finished" podID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerID="928a09032ef8dac95cf573e67859bfad56e0377885d8a1ec41b4be70c5a5ba3b" exitCode=0 Oct 30 22:04:39.340968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.339822 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" event={"ID":"06e69c90-2004-4bf2-94ac-a5030d369ac6","Type":"ContainerDied","Data":"928a09032ef8dac95cf573e67859bfad56e0377885d8a1ec41b4be70c5a5ba3b"} Oct 30 22:04:39.340968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.339873 4859 scope.go:122] "RemoveContainer" containerID="8e30beef849efec90473f50f0e8531fbb756aadb48d9d34fa55bbc4b0d980e74" Oct 30 22:04:39.359200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.359166 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq" Oct 30 22:04:39.534336 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.534306 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 22:04:39.705923 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.705889 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-kube-api-access-zklll\" (UniqueName: \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-kube-api-access-zklll\") pod \"06e69c90-2004-4bf2-94ac-a5030d369ac6\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " Oct 30 22:04:39.706172 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.706011 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-token\" (UniqueName: \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-token\") pod \"06e69c90-2004-4bf2-94ac-a5030d369ac6\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " Oct 30 22:04:39.706172 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.706084 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-certs\") pod \"06e69c90-2004-4bf2-94ac-a5030d369ac6\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " Oct 30 22:04:39.706172 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.706118 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-socket\") pod \"06e69c90-2004-4bf2-94ac-a5030d369ac6\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " Oct 30 22:04:39.706172 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.706146 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-podinfo\") pod \"06e69c90-2004-4bf2-94ac-a5030d369ac6\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " Oct 30 22:04:39.706407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.706213 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-envoy\") pod \"06e69c90-2004-4bf2-94ac-a5030d369ac6\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " Oct 30 22:04:39.706407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.706256 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-credential-socket\") pod \"06e69c90-2004-4bf2-94ac-a5030d369ac6\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " Oct 30 22:04:39.706407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.706325 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e69c90-2004-4bf2-94ac-a5030d369ac6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e69c90-2004-4bf2-94ac-a5030d369ac6-istiod-ca-cert\") pod \"06e69c90-2004-4bf2-94ac-a5030d369ac6\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " Oct 30 22:04:39.706575 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.706381 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-data\") pod \"06e69c90-2004-4bf2-94ac-a5030d369ac6\" (UID: \"06e69c90-2004-4bf2-94ac-a5030d369ac6\") " Oct 30 22:04:39.706907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.706880 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-credential-socket" pod "06e69c90-2004-4bf2-94ac-a5030d369ac6" (UID: "06e69c90-2004-4bf2-94ac-a5030d369ac6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:39.707408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.707384 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e69c90-2004-4bf2-94ac-a5030d369ac6-istiod-ca-cert" pod "06e69c90-2004-4bf2-94ac-a5030d369ac6" (UID: "06e69c90-2004-4bf2-94ac-a5030d369ac6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:39.707849 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.707562 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-certs" pod "06e69c90-2004-4bf2-94ac-a5030d369ac6" (UID: "06e69c90-2004-4bf2-94ac-a5030d369ac6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:39.707990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.707732 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-data" pod "06e69c90-2004-4bf2-94ac-a5030d369ac6" (UID: "06e69c90-2004-4bf2-94ac-a5030d369ac6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:39.708099 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.707807 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-socket" pod "06e69c90-2004-4bf2-94ac-a5030d369ac6" (UID: "06e69c90-2004-4bf2-94ac-a5030d369ac6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:39.709878 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.709848 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-kube-api-access-zklll" pod "06e69c90-2004-4bf2-94ac-a5030d369ac6" (UID: "06e69c90-2004-4bf2-94ac-a5030d369ac6"). InnerVolumeSpecName "kube-api-access-zklll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:39.712328 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.712292 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-token" pod "06e69c90-2004-4bf2-94ac-a5030d369ac6" (UID: "06e69c90-2004-4bf2-94ac-a5030d369ac6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:39.714058 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.712334 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-podinfo" pod "06e69c90-2004-4bf2-94ac-a5030d369ac6" (UID: "06e69c90-2004-4bf2-94ac-a5030d369ac6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:39.714058 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.713768 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-envoy" pod "06e69c90-2004-4bf2-94ac-a5030d369ac6" (UID: "06e69c90-2004-4bf2-94ac-a5030d369ac6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:39.808021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.807980 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.808021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.808020 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.808214 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.808033 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.808214 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.808042 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.808214 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.808051 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.808214 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.808060 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e69c90-2004-4bf2-94ac-a5030d369ac6-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.808214 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.808069 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.808214 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.808078 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zklll\" (UniqueName: \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-kube-api-access-zklll\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.808214 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:39.808089 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e69c90-2004-4bf2-94ac-a5030d369ac6-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.883309 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:04:39.883262 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:04:40.347287 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:40.347253 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" Oct 30 22:04:40.347287 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:40.347274 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g" event={"ID":"06e69c90-2004-4bf2-94ac-a5030d369ac6","Type":"ContainerDied","Data":"a8297314b53d8c6247bfc1d12a4d518628832898b7cbe56fba32e2e6d55f016a"} Oct 30 22:04:40.347798 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:40.347325 4859 scope.go:122] "RemoveContainer" containerID="928a09032ef8dac95cf573e67859bfad56e0377885d8a1ec41b4be70c5a5ba3b" Oct 30 22:04:40.513381 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:40.513349 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g"] Oct 30 22:04:40.563081 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:40.563052 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-5r74g"] Oct 30 22:04:40.744751 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:40.744676 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" path="/var/lib/kubelet/pods/06e69c90-2004-4bf2-94ac-a5030d369ac6/volumes" Oct 30 22:04:41.108307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.108277 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv"] Oct 30 22:04:41.108870 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.108858 4859 cpu_manager.go:414] "containerMap: removing container" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" Oct 30 22:04:41.108916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.108872 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" Oct 30 22:04:41.109022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.109012 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" Oct 30 22:04:41.113587 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.113573 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:04:41.125332 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.125310 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6f86b6-0960-4c9c-a022-3520ce8f51be-plugins-config-volume\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv\" (UID: \"3c6f86b6-0960-4c9c-a022-3520ce8f51be\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:04:41.125437 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.125381 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s58m\" (UniqueName: \"kubernetes.io/projected/3c6f86b6-0960-4c9c-a022-3520ce8f51be-kube-api-access-4s58m\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv\" (UID: \"3c6f86b6-0960-4c9c-a022-3520ce8f51be\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:04:41.167179 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.167144 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7fabe91d\"" Oct 30 22:04:41.167438 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.167249 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7fabe91d-dockercfg-r564w\"" Oct 30 22:04:41.178338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.178305 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv"] Oct 30 22:04:41.231375 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.229036 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6f86b6-0960-4c9c-a022-3520ce8f51be-plugins-config-volume\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv\" (UID: \"3c6f86b6-0960-4c9c-a022-3520ce8f51be\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:04:41.231375 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.229173 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6f86b6-0960-4c9c-a022-3520ce8f51be-plugins-config-volume\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv\" (UID: \"3c6f86b6-0960-4c9c-a022-3520ce8f51be\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:04:41.231767 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.231740 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s58m\" (UniqueName: \"kubernetes.io/projected/3c6f86b6-0960-4c9c-a022-3520ce8f51be-kube-api-access-4s58m\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv\" (UID: \"3c6f86b6-0960-4c9c-a022-3520ce8f51be\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:04:41.301693 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.301643 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s58m\" (UniqueName: \"kubernetes.io/projected/3c6f86b6-0960-4c9c-a022-3520ce8f51be-kube-api-access-4s58m\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv\" (UID: \"3c6f86b6-0960-4c9c-a022-3520ce8f51be\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:04:41.313673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.309676 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn"] Oct 30 22:04:41.313673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.310585 4859 cpu_manager.go:414] "containerMap: removing container" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" Oct 30 22:04:41.313673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.310603 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" Oct 30 22:04:41.313673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.310769 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e69c90-2004-4bf2-94ac-a5030d369ac6" containerName="istio-proxy" Oct 30 22:04:41.316028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.314930 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:04:41.336741 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.334462 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51abe81b-edcf-4513-be2a-7dd098b58e94-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn\" (UID: \"51abe81b-edcf-4513-be2a-7dd098b58e94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:04:41.336741 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.334644 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94bk\" (UniqueName: \"kubernetes.io/projected/51abe81b-edcf-4513-be2a-7dd098b58e94-kube-api-access-t94bk\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn\" (UID: \"51abe81b-edcf-4513-be2a-7dd098b58e94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:04:41.370033 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.365114 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn"] Oct 30 22:04:41.427367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.427062 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:04:41.435869 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.435836 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94bk\" (UniqueName: \"kubernetes.io/projected/51abe81b-edcf-4513-be2a-7dd098b58e94-kube-api-access-t94bk\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn\" (UID: \"51abe81b-edcf-4513-be2a-7dd098b58e94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:04:41.436032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.435905 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51abe81b-edcf-4513-be2a-7dd098b58e94-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn\" (UID: \"51abe81b-edcf-4513-be2a-7dd098b58e94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:04:41.436549 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.436530 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51abe81b-edcf-4513-be2a-7dd098b58e94-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn\" (UID: \"51abe81b-edcf-4513-be2a-7dd098b58e94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:04:41.505291 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.492326 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94bk\" (UniqueName: \"kubernetes.io/projected/51abe81b-edcf-4513-be2a-7dd098b58e94-kube-api-access-t94bk\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn\" (UID: \"51abe81b-edcf-4513-be2a-7dd098b58e94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:04:41.642465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.642376 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:04:41.809852 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:41.809825 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv"] Oct 30 22:04:41.811330 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:04:41.811304 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6f86b6_0960_4c9c_a022_3520ce8f51be.slice/crio-318c3b72e56f28e3708727d4b97fbe6e0b9072b0f621b4948944de789e1deb4a WatchSource:0}: Error finding container 318c3b72e56f28e3708727d4b97fbe6e0b9072b0f621b4948944de789e1deb4a: Status 404 returned error can't find the container with id 318c3b72e56f28e3708727d4b97fbe6e0b9072b0f621b4948944de789e1deb4a Oct 30 22:04:42.023887 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:42.023857 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn"] Oct 30 22:04:42.024743 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:04:42.024718 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51abe81b_edcf_4513_be2a_7dd098b58e94.slice/crio-2a40cb35c4d0447437e515f7ae912ab0fc39ec4bab720b39507578d8dfec4cfe WatchSource:0}: Error finding container 2a40cb35c4d0447437e515f7ae912ab0fc39ec4bab720b39507578d8dfec4cfe: Status 404 returned error can't find the container with id 2a40cb35c4d0447437e515f7ae912ab0fc39ec4bab720b39507578d8dfec4cfe Oct 30 22:04:42.365843 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:42.365805 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" event={"ID":"51abe81b-edcf-4513-be2a-7dd098b58e94","Type":"ContainerStarted","Data":"5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0"} Oct 30 22:04:42.366036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:42.365902 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" event={"ID":"51abe81b-edcf-4513-be2a-7dd098b58e94","Type":"ContainerStarted","Data":"2a40cb35c4d0447437e515f7ae912ab0fc39ec4bab720b39507578d8dfec4cfe"} Oct 30 22:04:42.366036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:42.365934 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:04:42.367640 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:42.367613 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" event={"ID":"3c6f86b6-0960-4c9c-a022-3520ce8f51be","Type":"ContainerStarted","Data":"c0642649149cf11adc64ffcd983eac64bac6b3122f0c3dd61aafe9860ae648e2"} Oct 30 22:04:42.367763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:42.367659 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" event={"ID":"3c6f86b6-0960-4c9c-a022-3520ce8f51be","Type":"ContainerStarted","Data":"318c3b72e56f28e3708727d4b97fbe6e0b9072b0f621b4948944de789e1deb4a"} Oct 30 22:04:42.367763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:42.367687 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:04:42.486479 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:42.486433 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" podStartSLOduration=1.486419314 podStartE2EDuration="1.486419314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:42.481551482 +0000 UTC m=+1339.854178224" watchObservedRunningTime="2026-10-30 22:04:42.486419314 +0000 UTC m=+1339.859046061" Oct 30 22:04:42.598914 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:42.598848 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" podStartSLOduration=1.5988314479999999 podStartE2EDuration="1.598831448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:42.594569513 +0000 UTC m=+1339.967196260" watchObservedRunningTime="2026-10-30 22:04:42.598831448 +0000 UTC m=+1339.971458195" Oct 30 22:04:47.866501 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:47.866466 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:04:53.377780 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:53.377747 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:04:53.378323 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:04:53.377815 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:05:03.612610 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.612573 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28"] Oct 30 22:05:03.619137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.619111 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.671726 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.671683 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-90e0d5-openshift-default-dockercfg-mfg6n\"" Oct 30 22:05:03.727462 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.727423 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28"] Oct 30 22:05:03.765847 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.765800 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee846847-c342-4905-895f-d49a6199ffc2-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.766081 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.765934 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.766081 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.766059 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.766202 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.766184 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7km\" (UniqueName: \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-kube-api-access-xh7km\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.766267 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.766252 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.766315 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.766280 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.766376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.766360 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee846847-c342-4905-895f-d49a6199ffc2-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.766434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.766421 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.766486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.766453 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868019 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868060 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868090 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee846847-c342-4905-895f-d49a6199ffc2-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868123 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868463 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868141 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868463 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868296 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee846847-c342-4905-895f-d49a6199ffc2-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868463 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868343 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868463 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868382 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868665 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868525 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868665 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868585 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7km\" (UniqueName: \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-kube-api-access-xh7km\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868767 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868584 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868827 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868793 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.868827 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.868816 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.869321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.869303 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee846847-c342-4905-895f-d49a6199ffc2-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.870770 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.870746 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.870881 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.870833 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee846847-c342-4905-895f-d49a6199ffc2-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.927491 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.927448 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.927650 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.927522 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7km\" (UniqueName: \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-kube-api-access-xh7km\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:03.935361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:03.935334 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:04.373921 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:04.373885 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28"] Oct 30 22:05:04.409734 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:05:04.409705 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee846847_c342_4905_895f_d49a6199ffc2.slice/crio-f04ab3e4fd8da73aad1719a7b15fdaa4aef2574154ba6465ad499f5719c5ce99 WatchSource:0}: Error finding container f04ab3e4fd8da73aad1719a7b15fdaa4aef2574154ba6465ad499f5719c5ce99: Status 404 returned error can't find the container with id f04ab3e4fd8da73aad1719a7b15fdaa4aef2574154ba6465ad499f5719c5ce99 Oct 30 22:05:04.412160 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:04.412127 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:04.412242 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:04.412194 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:04.412242 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:04.412221 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:04.615837 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:04.615782 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" event={"ID":"ee846847-c342-4905-895f-d49a6199ffc2","Type":"ContainerStarted","Data":"89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a"} Oct 30 22:05:04.616342 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:04.615872 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" event={"ID":"ee846847-c342-4905-895f-d49a6199ffc2","Type":"ContainerStarted","Data":"f04ab3e4fd8da73aad1719a7b15fdaa4aef2574154ba6465ad499f5719c5ce99"} Oct 30 22:05:04.738089 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:04.737985 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" podStartSLOduration=1.7379682760000001 podStartE2EDuration="1.737968276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:04.733452916 +0000 UTC m=+1362.106079662" watchObservedRunningTime="2026-10-30 22:05:04.737968276 +0000 UTC m=+1362.110595028" Oct 30 22:05:04.935605 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:04.935570 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:05.938096 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:05.938064 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:05:05.938464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:05.938115 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" podUID="ee846847-c342-4905-895f-d49a6199ffc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:05:06.936376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:06.936338 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:05:06.936556 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:06.936398 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" podUID="ee846847-c342-4905-895f-d49a6199ffc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:05:07.936176 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:07.936142 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:05:07.936595 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:07.936206 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" podUID="ee846847-c342-4905-895f-d49a6199ffc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:05:08.935994 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:08.935936 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:05:08.936167 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:08.936012 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" podUID="ee846847-c342-4905-895f-d49a6199ffc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:05:09.193677 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.193583 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj"] Oct 30 22:05:09.198131 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.198110 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.244407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.244365 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e65e971-7f9c-4007-822c-243c847822eb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.244590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.244434 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e65e971-7f9c-4007-822c-243c847822eb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.244590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.244542 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e65e971-7f9c-4007-822c-243c847822eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.244694 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.244648 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e65e971-7f9c-4007-822c-243c847822eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.244694 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.244683 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8bnq\" (UniqueName: \"kubernetes.io/projected/8e65e971-7f9c-4007-822c-243c847822eb-kube-api-access-m8bnq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.244784 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.244712 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e65e971-7f9c-4007-822c-243c847822eb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.244829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.244804 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e65e971-7f9c-4007-822c-243c847822eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.250300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.250265 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:05:09.283965 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.281126 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:05:09.283965 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.281266 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4nvfx\"" Oct 30 22:05:09.283965 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.281299 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:05:09.304195 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.304163 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj"] Oct 30 22:05:09.346717 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.346681 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e65e971-7f9c-4007-822c-243c847822eb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.346912 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.346756 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e65e971-7f9c-4007-822c-243c847822eb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.346912 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.346813 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e65e971-7f9c-4007-822c-243c847822eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.346912 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.346876 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e65e971-7f9c-4007-822c-243c847822eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.347112 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.346912 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8bnq\" (UniqueName: \"kubernetes.io/projected/8e65e971-7f9c-4007-822c-243c847822eb-kube-api-access-m8bnq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.347112 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.346966 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e65e971-7f9c-4007-822c-243c847822eb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.347112 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.347017 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e65e971-7f9c-4007-822c-243c847822eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.348031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.348004 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e65e971-7f9c-4007-822c-243c847822eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.349633 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.349603 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e65e971-7f9c-4007-822c-243c847822eb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.352273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.352254 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e65e971-7f9c-4007-822c-243c847822eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.352590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.352561 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e65e971-7f9c-4007-822c-243c847822eb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.357251 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.357228 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e65e971-7f9c-4007-822c-243c847822eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.403806 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.403774 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e65e971-7f9c-4007-822c-243c847822eb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.404160 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.404133 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8bnq\" (UniqueName: \"kubernetes.io/projected/8e65e971-7f9c-4007-822c-243c847822eb-kube-api-access-m8bnq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9rwj\" (UID: \"8e65e971-7f9c-4007-822c-243c847822eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.512403 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.512302 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:09.903330 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:05:09.903288 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:05:09.936490 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.936450 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:05:09.936663 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.936527 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" podUID="ee846847-c342-4905-895f-d49a6199ffc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:05:09.994355 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.994323 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj"] Oct 30 22:05:09.996332 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:05:09.996304 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e65e971_7f9c_4007_822c_243c847822eb.slice/crio-f081917735816efd6848fed268a64e1698b46434e5e9ada8fe5ec8319afa4dc3 WatchSource:0}: Error finding container f081917735816efd6848fed268a64e1698b46434e5e9ada8fe5ec8319afa4dc3: Status 404 returned error can't find the container with id f081917735816efd6848fed268a64e1698b46434e5e9ada8fe5ec8319afa4dc3 Oct 30 22:05:09.998546 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:09.998528 4859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:05:10.665006 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:10.664967 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" event={"ID":"8e65e971-7f9c-4007-822c-243c847822eb","Type":"ContainerStarted","Data":"f081917735816efd6848fed268a64e1698b46434e5e9ada8fe5ec8319afa4dc3"} Oct 30 22:05:10.936642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:10.936548 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:05:10.936642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:10.936621 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" podUID="ee846847-c342-4905-895f-d49a6199ffc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:05:11.941291 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:11.941261 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:11.941776 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:11.941648 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:11.942406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:11.942386 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:05:13.879045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:13.879013 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:13.879379 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:13.879095 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:14.715902 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:14.715866 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" event={"ID":"8e65e971-7f9c-4007-822c-243c847822eb","Type":"ContainerStarted","Data":"4e95e48885368a9f52e2b5817c8e8171da43bd8c166b8cf62e25aedf17515815"} Oct 30 22:05:14.716106 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:14.715975 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:14.717681 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:14.717656 4859 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c9rwj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:05:14.717785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:14.717700 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" podUID="8e65e971-7f9c-4007-822c-243c847822eb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:05:14.924043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:14.923987 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" podStartSLOduration=2.045541897 podStartE2EDuration="5.923971774s" totalImagesPullingTime="3.878429877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:14.882644181 +0000 UTC m=+1372.255270927" watchObservedRunningTime="2026-10-30 22:05:14.923971774 +0000 UTC m=+1372.296598514" Oct 30 22:05:15.723414 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:15.723388 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9rwj" Oct 30 22:05:16.467927 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:16.467888 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:05:19.888128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:19.888086 4859 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:20.015972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:20.015039 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mgw8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:20.015972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:20.015120 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mgw8n" podUID="b6e11781-e771-4f4d-8458-3966ee881c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:20.358121 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:20.358076 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:20.408114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:20.408070 4859 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:20.737175 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:20.737073 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" podUID="ec42687d-7e39-4655-8873-8f790f18baf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:21.068417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:21.067639 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:21.176495 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:21.175112 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6sd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:21.176764 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:21.176735 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:21.183572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:21.183544 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:21.183819 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:21.183796 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:39.898508 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:05:39.898412 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:05:48.296179 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:48.296146 4859 generic.go:356] "Generic (PLEG): container finished" podID="6373d52e-4048-41f8-a998-1532d9ed4403" containerID="a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269" exitCode=0 Oct 30 22:05:48.296686 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:48.296221 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" event={"ID":"6373d52e-4048-41f8-a998-1532d9ed4403","Type":"ContainerDied","Data":"a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269"} Oct 30 22:05:48.296686 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:48.296653 4859 scope.go:122] "RemoveContainer" containerID="a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269" Oct 30 22:05:48.298128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:48.298098 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:48.298232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:48.298181 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:48.298293 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:48.298270 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:49.309977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:49.309910 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" event={"ID":"6373d52e-4048-41f8-a998-1532d9ed4403","Type":"ContainerStarted","Data":"02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c"} Oct 30 22:05:49.954668 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:49.954636 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:05:49.956021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:49.955995 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:05:49.956145 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:49.956049 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:05:50.954854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:50.954813 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:05:50.955368 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:50.954870 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:05:51.955525 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:51.955484 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:05:51.956079 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:51.955595 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:05:52.955300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:52.955268 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:05:52.955496 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:52.955324 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:05:53.448475 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:53.448446 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:05:53.955139 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:53.955107 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:05:53.955307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:53.955170 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:05:54.641181 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:54.641146 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:05:54.960153 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:54.960080 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:05:54.961166 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:05:54.961145 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:06:09.876893 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:06:09.876858 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:06:19.215615 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.215576 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k"] Oct 30 22:06:19.218667 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.218649 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.328168 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.328138 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k"] Oct 30 22:06:19.406054 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.406019 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.406242 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.406178 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.406242 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.406231 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e4a7cd-72c3-447c-94ac-604279c051aa-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.406365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.406258 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.406365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.406277 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.406365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.406350 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.406490 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.406457 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.406538 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.406528 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.406574 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.406547 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7pg\" (UniqueName: \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-kube-api-access-5k7pg\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.508406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.508323 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.508406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.508357 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7pg\" (UniqueName: \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-kube-api-access-5k7pg\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.508593 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.508426 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.508593 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.508547 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.508593 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.508574 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e4a7cd-72c3-447c-94ac-604279c051aa-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.508744 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.508608 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.508744 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.508637 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.508744 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.508701 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.508898 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.508791 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.508994 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.508906 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.509132 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.509103 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.509242 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.509146 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.509488 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.509470 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e4a7cd-72c3-447c-94ac-604279c051aa-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.509847 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.509826 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.511198 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.511163 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.511417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.511399 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.564812 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.564774 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7pg\" (UniqueName: \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-kube-api-access-5k7pg\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.564984 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.564836 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:19.831313 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:19.831229 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:20.265960 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:20.265912 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k"] Oct 30 22:06:20.301695 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:06:20.301668 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e4a7cd_72c3_447c_94ac_604279c051aa.slice/crio-860b2b6d4d47ef6eaff2ae7b4dc84a8730b8dfc67cd5ffb3a7861f3ef9479c05 WatchSource:0}: Error finding container 860b2b6d4d47ef6eaff2ae7b4dc84a8730b8dfc67cd5ffb3a7861f3ef9479c05: Status 404 returned error can't find the container with id 860b2b6d4d47ef6eaff2ae7b4dc84a8730b8dfc67cd5ffb3a7861f3ef9479c05 Oct 30 22:06:20.304216 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:20.304185 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:06:20.304299 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:20.304258 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:06:20.304299 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:20.304287 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:06:20.574678 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:20.574581 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" event={"ID":"08e4a7cd-72c3-447c-94ac-604279c051aa","Type":"ContainerStarted","Data":"28793307505ac3efdf69d16e1b95351fad8d4109f142ab77e4351bf511fb39f1"} Oct 30 22:06:20.574847 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:20.574680 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" event={"ID":"08e4a7cd-72c3-447c-94ac-604279c051aa","Type":"ContainerStarted","Data":"860b2b6d4d47ef6eaff2ae7b4dc84a8730b8dfc67cd5ffb3a7861f3ef9479c05"} Oct 30 22:06:20.698407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:20.698354 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" podStartSLOduration=1.698334484 podStartE2EDuration="1.698334484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:20.696585093 +0000 UTC m=+1438.069211857" watchObservedRunningTime="2026-10-30 22:06:20.698334484 +0000 UTC m=+1438.070961229" Oct 30 22:06:20.832068 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:20.831918 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:21.833570 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:21.833539 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:06:21.834064 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:21.833589 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:06:22.832435 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:22.832401 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:06:22.832586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:22.832465 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:06:23.832382 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:23.832349 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:06:23.832760 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:23.832401 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:06:24.832043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:24.832011 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:06:24.832265 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:24.832243 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:06:25.832371 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:25.832330 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:06:25.832848 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:25.832393 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:06:26.831471 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:26.831442 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:06:26.831660 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:26.831499 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:06:27.839619 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:27.839583 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:27.840929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:27.840907 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:27.841742 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:27.841718 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:06:37.582621 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.582585 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt"] Oct 30 22:06:37.586055 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.586024 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.640854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.640829 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-da0f3d-serving-cert\"" Oct 30 22:06:37.650899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.650849 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt"] Oct 30 22:06:37.706450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.706414 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-metrics-certs\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.706596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.706476 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c33609-8347-4c22-bdb9-3201b4785ccc-openshift-service-ca\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.706657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.706607 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-maas-api-tls\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.706657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.706649 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rvv\" (UniqueName: \"kubernetes.io/projected/a2c33609-8347-4c22-bdb9-3201b4785ccc-kube-api-access-v9rvv\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.808518 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.808482 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-metrics-certs\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.808703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.808531 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c33609-8347-4c22-bdb9-3201b4785ccc-openshift-service-ca\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.808703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.808575 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-maas-api-tls\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.808703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.808598 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rvv\" (UniqueName: \"kubernetes.io/projected/a2c33609-8347-4c22-bdb9-3201b4785ccc-kube-api-access-v9rvv\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.808871 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:06:37.808732 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-serving-cert: secret "maas-api-e2e-api-a-da0f3d-serving-cert" not found Oct 30 22:06:37.808871 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:06:37.808807 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-maas-api-tls podName:a2c33609-8347-4c22-bdb9-3201b4785ccc nodeName:}" failed. No retries permitted until 2026-10-30 22:06:38.308787143 +0000 UTC m=+1455.681413877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-maas-api-tls") pod "maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" (UID: "a2c33609-8347-4c22-bdb9-3201b4785ccc") : secret "maas-api-e2e-api-a-da0f3d-serving-cert" not found Oct 30 22:06:37.809476 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.809451 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c33609-8347-4c22-bdb9-3201b4785ccc-openshift-service-ca\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.810922 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.810899 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-metrics-certs\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:37.867672 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:37.867637 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rvv\" (UniqueName: \"kubernetes.io/projected/a2c33609-8347-4c22-bdb9-3201b4785ccc-kube-api-access-v9rvv\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:38.316746 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:38.316659 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-maas-api-tls\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:38.319170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:38.319143 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-maas-api-tls\") pod \"maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:38.498603 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:38.498571 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:38.877835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:38.877802 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt"] Oct 30 22:06:38.878356 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:06:38.878197 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c33609_8347_4c22_bdb9_3201b4785ccc.slice/crio-453eea80dc2fe292595591f4b85fe529d8f461d533df1ba384f14580f63fd650 WatchSource:0}: Error finding container 453eea80dc2fe292595591f4b85fe529d8f461d533df1ba384f14580f63fd650: Status 404 returned error can't find the container with id 453eea80dc2fe292595591f4b85fe529d8f461d533df1ba384f14580f63fd650 Oct 30 22:06:39.740906 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:39.740847 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" event={"ID":"a2c33609-8347-4c22-bdb9-3201b4785ccc","Type":"ContainerStarted","Data":"453eea80dc2fe292595591f4b85fe529d8f461d533df1ba384f14580f63fd650"} Oct 30 22:06:39.877280 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:06:39.877235 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:06:40.755040 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:40.754997 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" event={"ID":"a2c33609-8347-4c22-bdb9-3201b4785ccc","Type":"ContainerStarted","Data":"50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4"} Oct 30 22:06:40.755461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:40.755107 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:06:40.874488 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:40.874123 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" podStartSLOduration=3.089635485 podStartE2EDuration="3.874105078s" totalImagesPullingTime="784.469593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:40.871364982 +0000 UTC m=+1458.243991726" watchObservedRunningTime="2026-10-30 22:06:40.874105078 +0000 UTC m=+1458.246731829" Oct 30 22:06:41.779520 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:41.779489 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:06:46.768625 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:06:46.768596 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:07:04.561072 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.561034 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg"] Oct 30 22:07:04.566801 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.566775 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.639556 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.639521 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbmnt\" (UniqueName: \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-kube-api-access-mbmnt\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.639740 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.639583 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.639740 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.639722 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.639833 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.639800 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.639883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.639830 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.639926 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.639906 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.639993 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.639939 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.639993 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.639970 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.640139 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.640126 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847c5aab-e0f2-4d3f-8484-77549e797e75-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.741306 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.741256 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.741306 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.741308 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.741600 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.741358 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.741600 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.741397 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.741600 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.741416 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.741600 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.741452 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847c5aab-e0f2-4d3f-8484-77549e797e75-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.741600 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.741568 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbmnt\" (UniqueName: \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-kube-api-access-mbmnt\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.742886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.741614 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.742886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.741660 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.742886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.742129 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.742886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.742664 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.742886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.742860 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.743639 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.743616 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847c5aab-e0f2-4d3f-8484-77549e797e75-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.743867 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.743845 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.745848 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.745826 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.746518 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.746491 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.800609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.800574 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.801302 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.801272 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbmnt\" (UniqueName: \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-kube-api-access-mbmnt\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:04.819585 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.819500 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg"] Oct 30 22:07:04.881257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:04.881222 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:05.326472 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:05.326443 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg"] Oct 30 22:07:05.326727 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:07:05.326706 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847c5aab_e0f2_4d3f_8484_77549e797e75.slice/crio-81e7e3e9a3ee3b0b811ea41e59e06fe96c7bca956884918058c05665d8d69525 WatchSource:0}: Error finding container 81e7e3e9a3ee3b0b811ea41e59e06fe96c7bca956884918058c05665d8d69525: Status 404 returned error can't find the container with id 81e7e3e9a3ee3b0b811ea41e59e06fe96c7bca956884918058c05665d8d69525 Oct 30 22:07:05.329441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:05.329407 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:05.329508 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:05.329474 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:05.329508 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:05.329504 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:05.949905 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:05.949874 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" event={"ID":"847c5aab-e0f2-4d3f-8484-77549e797e75","Type":"ContainerStarted","Data":"56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33"} Oct 30 22:07:05.950442 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:05.949930 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" event={"ID":"847c5aab-e0f2-4d3f-8484-77549e797e75","Type":"ContainerStarted","Data":"81e7e3e9a3ee3b0b811ea41e59e06fe96c7bca956884918058c05665d8d69525"} Oct 30 22:07:06.173965 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:06.173385 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" podStartSLOduration=2.173365719 podStartE2EDuration="2.173365719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:06.115569949 +0000 UTC m=+1483.488196698" watchObservedRunningTime="2026-10-30 22:07:06.173365719 +0000 UTC m=+1483.545992467" Oct 30 22:07:06.889966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:06.883306 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:06.890174 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:06.890065 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:07:06.890174 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:06.890127 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:07:07.887970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:07.883381 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:07:07.887970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:07.883444 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:07:08.887087 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:08.886092 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:07:08.887087 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:08.886153 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:07:09.408723 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:09.408474 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:07:09.881820 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:09.881782 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:07:09.886196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:09.886151 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:07:09.903271 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:07:09.903227 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:07:10.895869 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:10.888368 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:07:10.895869 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:10.888441 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:07:11.882609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:11.882561 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:07:11.882841 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:11.882632 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:07:13.261135 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:13.261093 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:13.261622 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:13.261550 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:13.265657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:13.265629 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:07:21.051817 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.051773 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv"] Oct 30 22:07:21.096099 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.096029 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.181227 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.181192 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-02a8fa-openshift-default-dockercfg-ngf4v\"" Oct 30 22:07:21.202012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.193926 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-certs\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.202012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.194146 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-credential-socket\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.202012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.194190 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-socket\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.202012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.194231 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-podinfo\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.202012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.194259 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-token\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.202012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.194326 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-envoy\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.202012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.194451 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfc9d\" (UniqueName: \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-kube-api-access-tfc9d\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.202012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.194486 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istiod-ca-cert\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.202012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.194572 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-data\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.241738 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.241681 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv"] Oct 30 22:07:21.296662 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.296619 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-data\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.297531 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.297503 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-certs\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.297994 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.297974 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-credential-socket\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.298162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.298137 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-socket\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.298263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.298193 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-podinfo\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.298263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.297906 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-certs\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.298263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.298225 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-token\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.298553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.298523 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-credential-socket\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.299377 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.297324 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-data\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.299377 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.298975 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-envoy\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.299377 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.299118 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfc9d\" (UniqueName: \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-kube-api-access-tfc9d\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.299377 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.299146 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-socket\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.299377 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.299156 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istiod-ca-cert\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.300249 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.300226 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istiod-ca-cert\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.301974 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.301898 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-podinfo\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.303326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.303300 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-envoy\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.356700 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.356663 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-token\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.358876 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.358853 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfc9d\" (UniqueName: \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-kube-api-access-tfc9d\") pod \"e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.521874 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.521841 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:21.972185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:21.972076 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv"] Oct 30 22:07:22.009637 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:07:22.009601 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847babb5_a5d6_42ad_bf35_86b6620bc9d0.slice/crio-338300e231d0a90ef9c7d431c10cb3fb230658f7f2ed16b4d2bbea4fe5e3e1f1 WatchSource:0}: Error finding container 338300e231d0a90ef9c7d431c10cb3fb230658f7f2ed16b4d2bbea4fe5e3e1f1: Status 404 returned error can't find the container with id 338300e231d0a90ef9c7d431c10cb3fb230658f7f2ed16b4d2bbea4fe5e3e1f1 Oct 30 22:07:22.013412 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:22.013360 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:22.013538 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:22.013464 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:22.013538 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:22.013511 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:22.227854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:22.227751 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" event={"ID":"847babb5-a5d6-42ad-bf35-86b6620bc9d0","Type":"ContainerStarted","Data":"af64bd0dddd2ea9c050168cda973986d7f3eb775c759c45a23a13daee23dc2fa"} Oct 30 22:07:22.227854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:22.227825 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" event={"ID":"847babb5-a5d6-42ad-bf35-86b6620bc9d0","Type":"ContainerStarted","Data":"338300e231d0a90ef9c7d431c10cb3fb230658f7f2ed16b4d2bbea4fe5e3e1f1"} Oct 30 22:07:22.350417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:22.350349 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" podStartSLOduration=2.35032816 podStartE2EDuration="2.35032816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:22.347022881 +0000 UTC m=+1499.719649647" watchObservedRunningTime="2026-10-30 22:07:22.35032816 +0000 UTC m=+1499.722954907" Oct 30 22:07:22.522093 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:22.522016 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:23.018647 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:23.018619 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:07:23.024925 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:23.024902 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:07:23.027261 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:23.027242 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:07:23.032375 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:23.032355 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:07:23.528902 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:23.528867 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:24.256559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:24.256520 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:24.257760 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:24.257738 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:07:36.885578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.885535 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7"] Oct 30 22:07:36.908977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.903868 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:36.929433 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.924777 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7"] Oct 30 22:07:36.977968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.977644 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvvd2\" (UniqueName: \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-kube-api-access-wvvd2\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:36.977968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.977694 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-credential-socket\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:36.977968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.977745 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istiod-ca-cert\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:36.977968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.977816 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-podinfo\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:36.977968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.977835 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-envoy\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:36.977968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.977860 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-data\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:36.977968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.977911 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-certs\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:36.977968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.977940 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-socket\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:36.978700 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:36.978015 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-token\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.006581 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.006541 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-479363-openshift-default-dockercfg-vwwhw\"" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.080913 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istiod-ca-cert\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.081128 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-podinfo\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.081172 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-envoy\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.081216 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-data\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.081306 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-certs\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.081355 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-socket\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.081387 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-token\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.081509 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvvd2\" (UniqueName: \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-kube-api-access-wvvd2\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.081544 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-credential-socket\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.082081 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-credential-socket\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.087572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.086529 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istiod-ca-cert\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.089560 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.089498 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-certs\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.090159 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.090109 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-socket\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.091608 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.091554 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-data\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.095801 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.095743 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-podinfo\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.113712 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.113675 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-envoy\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.146921 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.146844 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-token\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.146921 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.146897 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvvd2\" (UniqueName: \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-kube-api-access-wvvd2\") pod \"e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.235887 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.235856 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:37.689282 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.687459 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7"] Oct 30 22:07:37.729257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.727459 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:37.729257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.727566 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:37.729257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:37.727617 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:38.452969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:38.452386 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" event={"ID":"bfad3673-95ee-4fd4-b984-240d47f2e5b5","Type":"ContainerStarted","Data":"53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e"} Oct 30 22:07:38.452969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:38.452467 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" event={"ID":"bfad3673-95ee-4fd4-b984-240d47f2e5b5","Type":"ContainerStarted","Data":"c3d56b043b602c30acbca1c327c68211d36b600d2eec2077a15b546ed3fa6708"} Oct 30 22:07:38.575936 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:38.575863 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" podStartSLOduration=2.575844306 podStartE2EDuration="2.575844306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:38.571440148 +0000 UTC m=+1515.944066897" watchObservedRunningTime="2026-10-30 22:07:38.575844306 +0000 UTC m=+1515.948471053" Oct 30 22:07:39.251967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:39.246482 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:39.272969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:39.269497 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:39.517683 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:39.517597 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:39.520627 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:07:39.520606 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:07:39.982452 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:07:39.982133 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:08:02.036765 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:02.036727 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:08:09.884078 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:08:09.884036 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:08:25.888529 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:25.888490 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb"] Oct 30 22:08:25.898217 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:25.898186 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:25.952250 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:25.952218 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-02a8fa-serving-cert\"" Oct 30 22:08:25.954173 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:25.954148 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a6df79-f777-4e98-b749-99a27e252613-openshift-service-ca\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:25.954285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:25.954190 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctjs\" (UniqueName: \"kubernetes.io/projected/b7a6df79-f777-4e98-b749-99a27e252613-kube-api-access-sctjs\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:25.954363 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:25.954342 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-maas-api-tls\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:25.954426 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:25.954402 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-metrics-certs\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:26.006578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.006541 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb"] Oct 30 22:08:26.056312 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.056276 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a6df79-f777-4e98-b749-99a27e252613-openshift-service-ca\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:26.056470 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.056350 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctjs\" (UniqueName: \"kubernetes.io/projected/b7a6df79-f777-4e98-b749-99a27e252613-kube-api-access-sctjs\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:26.056517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.056500 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-maas-api-tls\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:26.056573 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.056558 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-metrics-certs\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:26.056650 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:08:26.056630 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-serving-cert: secret "maas-api-e2e-api-b-02a8fa-serving-cert" not found Oct 30 22:08:26.056705 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:08:26.056697 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-maas-api-tls podName:b7a6df79-f777-4e98-b749-99a27e252613 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:26.556677787 +0000 UTC m=+1563.929304526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-maas-api-tls") pod "maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" (UID: "b7a6df79-f777-4e98-b749-99a27e252613") : secret "maas-api-e2e-api-b-02a8fa-serving-cert" not found Oct 30 22:08:26.056925 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.056901 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a6df79-f777-4e98-b749-99a27e252613-openshift-service-ca\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:26.058983 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.058931 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-metrics-certs\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:26.119331 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.119302 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctjs\" (UniqueName: \"kubernetes.io/projected/b7a6df79-f777-4e98-b749-99a27e252613-kube-api-access-sctjs\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:26.565938 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.565897 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-maas-api-tls\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:26.568373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.568348 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-maas-api-tls\") pod \"maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:26.823342 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:26.823253 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:27.229196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:27.229163 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb"] Oct 30 22:08:27.229669 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:08:27.229642 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a6df79_f777_4e98_b749_99a27e252613.slice/crio-79d77131aaa91895b1728351943b94eb3d05a15c828ce00bf49ce25eff8a4cbe WatchSource:0}: Error finding container 79d77131aaa91895b1728351943b94eb3d05a15c828ce00bf49ce25eff8a4cbe: Status 404 returned error can't find the container with id 79d77131aaa91895b1728351943b94eb3d05a15c828ce00bf49ce25eff8a4cbe Oct 30 22:08:27.933425 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:27.933383 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" event={"ID":"b7a6df79-f777-4e98-b749-99a27e252613","Type":"ContainerStarted","Data":"79d77131aaa91895b1728351943b94eb3d05a15c828ce00bf49ce25eff8a4cbe"} Oct 30 22:08:28.951893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:28.951857 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" event={"ID":"b7a6df79-f777-4e98-b749-99a27e252613","Type":"ContainerStarted","Data":"e31b2f8633574177e7d674d177c8ea8d8d567db4d47aec09dcb1234778471a87"} Oct 30 22:08:28.952321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:28.952026 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:29.080970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:29.073627 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" podStartSLOduration=3.292890246 podStartE2EDuration="4.073606609s" totalImagesPullingTime="780.716363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:29.070455423 +0000 UTC m=+1566.443082171" watchObservedRunningTime="2026-10-30 22:08:29.073606609 +0000 UTC m=+1566.446233360" Oct 30 22:08:34.964394 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:34.964322 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:08:36.031184 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.031151 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k"] Oct 30 22:08:36.035936 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.035911 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.089678 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.089653 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-479363-serving-cert\"" Oct 30 22:08:36.153227 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.153188 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k"] Oct 30 22:08:36.216026 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.215992 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d718ff5-697c-4936-8417-c4fe77e2ca6c-openshift-service-ca\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.216193 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.216029 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-maas-api-tls\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.216193 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.216117 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-metrics-certs\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.216411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.216316 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk255\" (UniqueName: \"kubernetes.io/projected/5d718ff5-697c-4936-8417-c4fe77e2ca6c-kube-api-access-gk255\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.318137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.318048 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d718ff5-697c-4936-8417-c4fe77e2ca6c-openshift-service-ca\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.318137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.318093 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-maas-api-tls\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.318137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.318133 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-metrics-certs\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.318399 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:08:36.318253 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-479363-serving-cert: secret "maas-api-e2e-api-b-479363-serving-cert" not found Oct 30 22:08:36.318399 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:08:36.318321 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-maas-api-tls podName:5d718ff5-697c-4936-8417-c4fe77e2ca6c nodeName:}" failed. No retries permitted until 2026-10-30 22:08:36.818303393 +0000 UTC m=+1574.190930131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-maas-api-tls") pod "maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" (UID: "5d718ff5-697c-4936-8417-c4fe77e2ca6c") : secret "maas-api-e2e-api-b-479363-serving-cert" not found Oct 30 22:08:36.318486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.318410 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk255\" (UniqueName: \"kubernetes.io/projected/5d718ff5-697c-4936-8417-c4fe77e2ca6c-kube-api-access-gk255\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.318870 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.318842 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d718ff5-697c-4936-8417-c4fe77e2ca6c-openshift-service-ca\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.320742 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.320718 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-metrics-certs\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.378638 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.378573 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk255\" (UniqueName: \"kubernetes.io/projected/5d718ff5-697c-4936-8417-c4fe77e2ca6c-kube-api-access-gk255\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.833466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.833430 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-maas-api-tls\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.845675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.845638 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-maas-api-tls\") pod \"maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:36.949547 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:36.949515 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:37.357845 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:08:37.357808 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d718ff5_697c_4936_8417_c4fe77e2ca6c.slice/crio-b5e25931310d09e0a59ce496755db7cb4cc202f7ca85422cb0666415234d36ba WatchSource:0}: Error finding container b5e25931310d09e0a59ce496755db7cb4cc202f7ca85422cb0666415234d36ba: Status 404 returned error can't find the container with id b5e25931310d09e0a59ce496755db7cb4cc202f7ca85422cb0666415234d36ba Oct 30 22:08:37.359346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:37.359315 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k"] Oct 30 22:08:38.026897 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:38.026863 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" event={"ID":"5d718ff5-697c-4936-8417-c4fe77e2ca6c","Type":"ContainerStarted","Data":"b5e25931310d09e0a59ce496755db7cb4cc202f7ca85422cb0666415234d36ba"} Oct 30 22:08:39.036054 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:39.036017 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" event={"ID":"5d718ff5-697c-4936-8417-c4fe77e2ca6c","Type":"ContainerStarted","Data":"47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7"} Oct 30 22:08:39.036466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:39.036084 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:39.159441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:39.159386 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" podStartSLOduration=3.354933411 podStartE2EDuration="4.159367107s" totalImagesPullingTime="804.433696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:39.156555878 +0000 UTC m=+1576.529182627" watchObservedRunningTime="2026-10-30 22:08:39.159367107 +0000 UTC m=+1576.531993861" Oct 30 22:08:39.882685 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:08:39.882642 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:08:43.773176 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:43.773139 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:08:45.048488 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:45.048456 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:08:54.891582 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:54.891546 4859 patch_prober.go:29] interesting pod/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": read tcp 10.132.0.2:52798->10.132.0.114:15021: read: connection reset by peer" start-of-body= Oct 30 22:08:54.892087 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:54.891627 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": read tcp 10.132.0.2:52798->10.132.0.114:15021: read: connection reset by peer" Oct 30 22:08:55.222699 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:55.222619 4859 generic.go:356] "Generic (PLEG): container finished" podID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerID="af64bd0dddd2ea9c050168cda973986d7f3eb775c759c45a23a13daee23dc2fa" exitCode=0 Oct 30 22:08:55.222837 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:55.222692 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" event={"ID":"847babb5-a5d6-42ad-bf35-86b6620bc9d0","Type":"ContainerDied","Data":"af64bd0dddd2ea9c050168cda973986d7f3eb775c759c45a23a13daee23dc2fa"} Oct 30 22:08:55.223134 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:55.223117 4859 scope.go:122] "RemoveContainer" containerID="af64bd0dddd2ea9c050168cda973986d7f3eb775c759c45a23a13daee23dc2fa" Oct 30 22:08:55.224584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:55.224557 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:08:55.224650 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:55.224622 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:08:55.224690 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:55.224663 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:08:56.231935 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:56.231897 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" event={"ID":"847babb5-a5d6-42ad-bf35-86b6620bc9d0","Type":"ContainerStarted","Data":"36f6c9341b4db0748e555d07d3e3a70e7d0ece230474705ecc49504782b9f817"} Oct 30 22:08:56.522439 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:56.522353 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:08:56.522604 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:56.522581 4859 patch_prober.go:29] interesting pod/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:08:56.522675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:56.522657 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 22:08:57.523132 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:57.523096 4859 patch_prober.go:29] interesting pod/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:08:57.523534 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:57.523154 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 22:08:58.524397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:58.523918 4859 patch_prober.go:29] interesting pod/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:08:58.524397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:58.524011 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 22:08:59.523043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:59.523009 4859 patch_prober.go:29] interesting pod/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:08:59.523223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:08:59.523062 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 22:09:00.522452 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:00.522413 4859 patch_prober.go:29] interesting pod/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:09:00.522891 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:00.522479 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 22:09:01.527652 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:01.527620 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:09:01.528095 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:01.528070 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:09:01.528710 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:01.528695 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:09:02.372075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:02.372042 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv"] Oct 30 22:09:02.493597 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:02.493564 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn"] Oct 30 22:09:07.372680 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:07.372627 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" podUID="3c6f86b6-0960-4c9c-a022-3520ce8f51be" containerName="payload-processing" containerID="cri-o://c0642649149cf11adc64ffcd983eac64bac6b3122f0c3dd61aafe9860ae648e2" gracePeriod=25 Oct 30 22:09:07.493998 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:07.493956 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" podUID="51abe81b-edcf-4513-be2a-7dd098b58e94" containerName="payload-pre-processing" containerID="cri-o://5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0" gracePeriod=25 Oct 30 22:09:09.879798 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:09:09.879756 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:09:11.179659 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:11.179618 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:09:13.387118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:13.387077 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" podUID="3c6f86b6-0960-4c9c-a022-3520ce8f51be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:09:13.387643 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:13.387222 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" podUID="51abe81b-edcf-4513-be2a-7dd098b58e94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:09:23.376360 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:23.376319 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" podUID="3c6f86b6-0960-4c9c-a022-3520ce8f51be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:09:23.376767 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:23.376319 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" podUID="51abe81b-edcf-4513-be2a-7dd098b58e94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:09:23.510129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:23.510095 4859 generic.go:356] "Generic (PLEG): container finished" podID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerID="53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e" exitCode=0 Oct 30 22:09:23.510298 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:23.510175 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" event={"ID":"bfad3673-95ee-4fd4-b984-240d47f2e5b5","Type":"ContainerDied","Data":"53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e"} Oct 30 22:09:23.510692 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:23.510672 4859 scope.go:122] "RemoveContainer" containerID="53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e" Oct 30 22:09:23.512404 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:23.512374 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:09:23.512516 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:23.512461 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:09:23.512583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:23.512511 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:09:24.515663 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:24.515627 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:09:24.523473 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:24.523443 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" event={"ID":"bfad3673-95ee-4fd4-b984-240d47f2e5b5","Type":"ContainerStarted","Data":"2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693"} Oct 30 22:09:25.236006 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:25.235976 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:09:25.236831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:25.236810 4859 patch_prober.go:29] interesting pod/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:09:25.236901 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:25.236864 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:09:26.237082 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:26.237044 4859 patch_prober.go:29] interesting pod/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:09:26.237670 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:26.237106 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:09:27.237171 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.237138 4859 patch_prober.go:29] interesting pod/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:09:27.237562 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.237190 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:09:27.549473 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.549443 4859 generic.go:356] "Generic (PLEG): container finished" podID="3c6f86b6-0960-4c9c-a022-3520ce8f51be" containerID="c0642649149cf11adc64ffcd983eac64bac6b3122f0c3dd61aafe9860ae648e2" exitCode=0 Oct 30 22:09:27.549632 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.549482 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" event={"ID":"3c6f86b6-0960-4c9c-a022-3520ce8f51be","Type":"ContainerDied","Data":"c0642649149cf11adc64ffcd983eac64bac6b3122f0c3dd61aafe9860ae648e2"} Oct 30 22:09:27.751587 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.751187 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:09:27.852874 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.852847 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6f86b6-0960-4c9c-a022-3520ce8f51be-kube-api-access-4s58m\" (UniqueName: \"kubernetes.io/projected/3c6f86b6-0960-4c9c-a022-3520ce8f51be-kube-api-access-4s58m\") pod \"3c6f86b6-0960-4c9c-a022-3520ce8f51be\" (UID: \"3c6f86b6-0960-4c9c-a022-3520ce8f51be\") " Oct 30 22:09:27.853254 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.853230 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6f86b6-0960-4c9c-a022-3520ce8f51be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6f86b6-0960-4c9c-a022-3520ce8f51be-plugins-config-volume\") pod \"3c6f86b6-0960-4c9c-a022-3520ce8f51be\" (UID: \"3c6f86b6-0960-4c9c-a022-3520ce8f51be\") " Oct 30 22:09:27.853618 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.853596 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6f86b6-0960-4c9c-a022-3520ce8f51be-plugins-config-volume" pod "3c6f86b6-0960-4c9c-a022-3520ce8f51be" (UID: "3c6f86b6-0960-4c9c-a022-3520ce8f51be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:27.853968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.853939 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6f86b6-0960-4c9c-a022-3520ce8f51be-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:27.855805 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.855772 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6f86b6-0960-4c9c-a022-3520ce8f51be-kube-api-access-4s58m" pod "3c6f86b6-0960-4c9c-a022-3520ce8f51be" (UID: "3c6f86b6-0960-4c9c-a022-3520ce8f51be"). InnerVolumeSpecName "kube-api-access-4s58m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:27.913369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.913344 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:09:27.955374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.955345 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51abe81b-edcf-4513-be2a-7dd098b58e94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51abe81b-edcf-4513-be2a-7dd098b58e94-plugins-config-volume\") pod \"51abe81b-edcf-4513-be2a-7dd098b58e94\" (UID: \"51abe81b-edcf-4513-be2a-7dd098b58e94\") " Oct 30 22:09:27.955540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.955526 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51abe81b-edcf-4513-be2a-7dd098b58e94-kube-api-access-t94bk\" (UniqueName: \"kubernetes.io/projected/51abe81b-edcf-4513-be2a-7dd098b58e94-kube-api-access-t94bk\") pod \"51abe81b-edcf-4513-be2a-7dd098b58e94\" (UID: \"51abe81b-edcf-4513-be2a-7dd098b58e94\") " Oct 30 22:09:27.955742 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.955698 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51abe81b-edcf-4513-be2a-7dd098b58e94-plugins-config-volume" pod "51abe81b-edcf-4513-be2a-7dd098b58e94" (UID: "51abe81b-edcf-4513-be2a-7dd098b58e94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:27.956021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.955987 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s58m\" (UniqueName: \"kubernetes.io/projected/3c6f86b6-0960-4c9c-a022-3520ce8f51be-kube-api-access-4s58m\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:27.956021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.956005 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51abe81b-edcf-4513-be2a-7dd098b58e94-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:27.957652 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:27.957630 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51abe81b-edcf-4513-be2a-7dd098b58e94-kube-api-access-t94bk" pod "51abe81b-edcf-4513-be2a-7dd098b58e94" (UID: "51abe81b-edcf-4513-be2a-7dd098b58e94"). InnerVolumeSpecName "kube-api-access-t94bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:28.057443 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.057368 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t94bk\" (UniqueName: \"kubernetes.io/projected/51abe81b-edcf-4513-be2a-7dd098b58e94-kube-api-access-t94bk\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:28.237047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.237013 4859 patch_prober.go:29] interesting pod/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:09:28.237215 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.237061 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:09:28.589967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.587639 4859 generic.go:356] "Generic (PLEG): container finished" podID="51abe81b-edcf-4513-be2a-7dd098b58e94" containerID="5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0" exitCode=0 Oct 30 22:09:28.589967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.587896 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" event={"ID":"51abe81b-edcf-4513-be2a-7dd098b58e94","Type":"ContainerDied","Data":"5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0"} Oct 30 22:09:28.589967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.587971 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" event={"ID":"51abe81b-edcf-4513-be2a-7dd098b58e94","Type":"ContainerDied","Data":"2a40cb35c4d0447437e515f7ae912ab0fc39ec4bab720b39507578d8dfec4cfe"} Oct 30 22:09:28.589967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.588006 4859 scope.go:122] "RemoveContainer" containerID="5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0" Oct 30 22:09:28.589967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.588152 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn" Oct 30 22:09:28.593997 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.593789 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" event={"ID":"3c6f86b6-0960-4c9c-a022-3520ce8f51be","Type":"ContainerDied","Data":"318c3b72e56f28e3708727d4b97fbe6e0b9072b0f621b4948944de789e1deb4a"} Oct 30 22:09:28.593997 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.593876 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv" Oct 30 22:09:28.606715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.606693 4859 scope.go:122] "RemoveContainer" containerID="5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0" Oct 30 22:09:28.607011 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:09:28.606985 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0\": container with ID starting with 5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0 not found: ID does not exist" containerID="5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0" Oct 30 22:09:28.607131 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.607016 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0"} err="failed to get container status \"5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0\": rpc error: code = NotFound desc = could not find container \"5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0\": container with ID starting with 5d228730842e0a20e7df2fe7a06a9b6bc1b3c5bc1ebb1b9c87d63d4a25f128b0 not found: ID does not exist" Oct 30 22:09:28.607131 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.607034 4859 scope.go:122] "RemoveContainer" containerID="c0642649149cf11adc64ffcd983eac64bac6b3122f0c3dd61aafe9860ae648e2" Oct 30 22:09:28.782025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.780990 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn"] Oct 30 22:09:28.814037 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.814003 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-tshgn"] Oct 30 22:09:28.889054 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.889019 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv"] Oct 30 22:09:28.899784 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:28.899740 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-csvrv"] Oct 30 22:09:29.236238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:29.236206 4859 patch_prober.go:29] interesting pod/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:09:29.236406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:29.236257 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:09:30.242338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:30.242307 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:09:30.242823 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:30.242806 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:09:30.243551 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:30.243497 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:09:30.742550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:30.742515 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6f86b6-0960-4c9c-a022-3520ce8f51be" path="/var/lib/kubelet/pods/3c6f86b6-0960-4c9c-a022-3520ce8f51be/volumes" Oct 30 22:09:30.743552 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:30.743528 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51abe81b-edcf-4513-be2a-7dd098b58e94" path="/var/lib/kubelet/pods/51abe81b-edcf-4513-be2a-7dd098b58e94/volumes" Oct 30 22:09:39.878339 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:09:39.878300 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:09:51.268939 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.268906 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976"] Oct 30 22:09:51.269569 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.269554 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3c6f86b6-0960-4c9c-a022-3520ce8f51be" containerName="payload-processing" Oct 30 22:09:51.269620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.269569 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6f86b6-0960-4c9c-a022-3520ce8f51be" containerName="payload-processing" Oct 30 22:09:51.269620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.269600 4859 cpu_manager.go:414] "containerMap: removing container" podUID="51abe81b-edcf-4513-be2a-7dd098b58e94" containerName="payload-pre-processing" Oct 30 22:09:51.269620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.269605 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51abe81b-edcf-4513-be2a-7dd098b58e94" containerName="payload-pre-processing" Oct 30 22:09:51.269729 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.269719 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="51abe81b-edcf-4513-be2a-7dd098b58e94" containerName="payload-pre-processing" Oct 30 22:09:51.269762 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.269736 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6f86b6-0960-4c9c-a022-3520ce8f51be" containerName="payload-processing" Oct 30 22:09:51.273265 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.273250 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.317329 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.317304 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.317435 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.317338 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d6pf\" (UniqueName: \"kubernetes.io/projected/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-kube-api-access-7d6pf\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.317496 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.317472 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-maas-api-tls\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.317561 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.317549 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-metrics-certs\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.377927 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.377901 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-97ca39-serving-cert\"" Oct 30 22:09:51.382058 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.382032 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976"] Oct 30 22:09:51.419064 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.419038 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.419180 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.419074 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d6pf\" (UniqueName: \"kubernetes.io/projected/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-kube-api-access-7d6pf\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.419278 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.419256 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-maas-api-tls\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.419359 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.419348 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-metrics-certs\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.419443 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:09:51.419387 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-serving-cert: secret "maas-api-e2e-ipp-a-97ca39-serving-cert" not found Oct 30 22:09:51.419483 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:09:51.419460 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-maas-api-tls podName:9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74 nodeName:}" failed. No retries permitted until 2026-10-30 22:09:51.919440032 +0000 UTC m=+1649.292066764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-maas-api-tls") pod "maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" (UID: "9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74") : secret "maas-api-e2e-ipp-a-97ca39-serving-cert" not found Oct 30 22:09:51.419776 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.419735 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.421702 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.421681 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-metrics-certs\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.477078 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.477047 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d6pf\" (UniqueName: \"kubernetes.io/projected/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-kube-api-access-7d6pf\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.927062 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.927027 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-maas-api-tls\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:51.930605 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:51.930575 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-maas-api-tls\") pod \"maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:52.186345 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:52.186263 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:52.573147 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:09:52.573115 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4aee3c_d4e7_47ef_9e8f_1dd2537ddb74.slice/crio-79f8137217ba5fac8c8da05cc107806077dae5f1baaf9bad927f45b59fd68c9a WatchSource:0}: Error finding container 79f8137217ba5fac8c8da05cc107806077dae5f1baaf9bad927f45b59fd68c9a: Status 404 returned error can't find the container with id 79f8137217ba5fac8c8da05cc107806077dae5f1baaf9bad927f45b59fd68c9a Oct 30 22:09:52.573865 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:52.573838 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976"] Oct 30 22:09:52.792017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:52.791982 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" event={"ID":"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74","Type":"ContainerStarted","Data":"79f8137217ba5fac8c8da05cc107806077dae5f1baaf9bad927f45b59fd68c9a"} Oct 30 22:09:53.802550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:53.802443 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" event={"ID":"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74","Type":"ContainerStarted","Data":"553a4bf35cdeff921ca39a9f3022c832a72eb115e55f2955581286ac4190cd80"} Oct 30 22:09:53.802550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:53.802517 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:09:53.921097 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:53.921042 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" podStartSLOduration=2.062319011 podStartE2EDuration="2.921027255s" totalImagesPullingTime="858.708244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:53.916666648 +0000 UTC m=+1651.289293420" watchObservedRunningTime="2026-10-30 22:09:53.921027255 +0000 UTC m=+1651.293654001" Oct 30 22:09:59.817790 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:09:59.817760 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:10:05.078158 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:05.078071 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk"] Oct 30 22:10:05.078602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:05.078325 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" containerID="cri-o://f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e" gracePeriod=30 Oct 30 22:10:09.880051 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:10:09.880006 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:10:10.226876 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.226841 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:10:10.227070 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.226918 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:10:10.303047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.303005 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57"] Oct 30 22:10:10.303360 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.303332 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" containerID="cri-o://a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9" gracePeriod=30 Oct 30 22:10:10.328349 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.328323 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p"] Oct 30 22:10:10.328686 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.328658 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" containerID="cri-o://8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a" gracePeriod=30 Oct 30 22:10:10.335231 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.335207 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k"] Oct 30 22:10:10.335531 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.335506 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" containerID="cri-o://2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267" gracePeriod=30 Oct 30 22:10:10.365754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.365730 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:10:10.380171 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.380143 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt"] Oct 30 22:10:10.380375 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.380353 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" containerID="cri-o://6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3" gracePeriod=30 Oct 30 22:10:10.559471 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.559327 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-envoy\") pod \"3479d7ff-6792-481a-ad14-0672fb6ff04e\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " Oct 30 22:10:10.559654 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.559511 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-podinfo\") pod \"3479d7ff-6792-481a-ad14-0672fb6ff04e\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " Oct 30 22:10:10.559654 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.559558 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-socket\") pod \"3479d7ff-6792-481a-ad14-0672fb6ff04e\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " Oct 30 22:10:10.559654 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.559681 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-credential-socket\") pod \"3479d7ff-6792-481a-ad14-0672fb6ff04e\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " Oct 30 22:10:10.559829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.559721 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-kube-api-access-9z857\" (UniqueName: \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-kube-api-access-9z857\") pod \"3479d7ff-6792-481a-ad14-0672fb6ff04e\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " Oct 30 22:10:10.559829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.559749 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-certs\") pod \"3479d7ff-6792-481a-ad14-0672fb6ff04e\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " Oct 30 22:10:10.559829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.559767 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-token\" (UniqueName: \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-token\") pod \"3479d7ff-6792-481a-ad14-0672fb6ff04e\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " Oct 30 22:10:10.559829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.559821 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3479d7ff-6792-481a-ad14-0672fb6ff04e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3479d7ff-6792-481a-ad14-0672fb6ff04e-istiod-ca-cert\") pod \"3479d7ff-6792-481a-ad14-0672fb6ff04e\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " Oct 30 22:10:10.560052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.559849 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-data\") pod \"3479d7ff-6792-481a-ad14-0672fb6ff04e\" (UID: \"3479d7ff-6792-481a-ad14-0672fb6ff04e\") " Oct 30 22:10:10.560421 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.560393 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-socket" pod "3479d7ff-6792-481a-ad14-0672fb6ff04e" (UID: "3479d7ff-6792-481a-ad14-0672fb6ff04e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:10.560421 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.560405 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-credential-socket" pod "3479d7ff-6792-481a-ad14-0672fb6ff04e" (UID: "3479d7ff-6792-481a-ad14-0672fb6ff04e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:10.560639 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.560619 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-certs" pod "3479d7ff-6792-481a-ad14-0672fb6ff04e" (UID: "3479d7ff-6792-481a-ad14-0672fb6ff04e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:10.560740 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.560720 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3479d7ff-6792-481a-ad14-0672fb6ff04e-istiod-ca-cert" pod "3479d7ff-6792-481a-ad14-0672fb6ff04e" (UID: "3479d7ff-6792-481a-ad14-0672fb6ff04e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:10.562350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.560987 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-data" pod "3479d7ff-6792-481a-ad14-0672fb6ff04e" (UID: "3479d7ff-6792-481a-ad14-0672fb6ff04e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:10.562350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.561261 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:10.562350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.561276 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:10.562350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.561285 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:10.562350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.561293 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3479d7ff-6792-481a-ad14-0672fb6ff04e-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:10.562350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.561301 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:10.563725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.563701 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-envoy" pod "3479d7ff-6792-481a-ad14-0672fb6ff04e" (UID: "3479d7ff-6792-481a-ad14-0672fb6ff04e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:10.564081 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.564058 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-kube-api-access-9z857" pod "3479d7ff-6792-481a-ad14-0672fb6ff04e" (UID: "3479d7ff-6792-481a-ad14-0672fb6ff04e"). InnerVolumeSpecName "kube-api-access-9z857". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:10.566315 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.566197 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-token" pod "3479d7ff-6792-481a-ad14-0672fb6ff04e" (UID: "3479d7ff-6792-481a-ad14-0672fb6ff04e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:10.566315 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.566245 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-podinfo" pod "3479d7ff-6792-481a-ad14-0672fb6ff04e" (UID: "3479d7ff-6792-481a-ad14-0672fb6ff04e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:10.662919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.662883 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z857\" (UniqueName: \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-kube-api-access-9z857\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:10.662919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.662913 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:10.662919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.662923 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:10.663265 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.662932 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3479d7ff-6792-481a-ad14-0672fb6ff04e-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:10.949062 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.949029 4859 generic.go:356] "Generic (PLEG): container finished" podID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerID="f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e" exitCode=0 Oct 30 22:10:10.949565 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.949091 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" event={"ID":"3479d7ff-6792-481a-ad14-0672fb6ff04e","Type":"ContainerDied","Data":"f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e"} Oct 30 22:10:10.949565 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.949137 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" event={"ID":"3479d7ff-6792-481a-ad14-0672fb6ff04e","Type":"ContainerDied","Data":"c32667dc9cdec4ca5af34b7901b04711bdb6099ca3f2c30e53fcfc52bba8a0dd"} Oct 30 22:10:10.949565 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.949161 4859 scope.go:122] "RemoveContainer" containerID="f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e" Oct 30 22:10:10.949565 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.949092 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk" Oct 30 22:10:10.965472 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.965447 4859 scope.go:122] "RemoveContainer" containerID="f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e" Oct 30 22:10:10.965800 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:10:10.965777 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e\": container with ID starting with f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e not found: ID does not exist" containerID="f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e" Oct 30 22:10:10.965887 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:10.965811 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e"} err="failed to get container status \"f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e\": rpc error: code = NotFound desc = could not find container \"f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e\": container with ID starting with f71414d8f5374f0b87490ff205ed923bfbbe6e0be6cbc92a1087382c6ddb3c4e not found: ID does not exist" Oct 30 22:10:11.116200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:11.116165 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk"] Oct 30 22:10:11.127815 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:11.127783 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-nsftk"] Oct 30 22:10:12.752465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:12.752429 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" path="/var/lib/kubelet/pods/3479d7ff-6792-481a-ad14-0672fb6ff04e/volumes" Oct 30 22:10:13.481046 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:13.481007 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:10:13.481228 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:13.481075 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:10:13.748493 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:13.748324 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:10:15.114372 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.114333 4859 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:10:15.114759 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.114407 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:10:15.684889 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.684862 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 22:10:15.748143 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.748111 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:10:15.753827 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.753797 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:10:15.758235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.758205 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:10:15.805494 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.805451 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-token\" (UniqueName: \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-token\") pod \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " Oct 30 22:10:15.806029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.805617 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-podinfo\") pod \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " Oct 30 22:10:15.806029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.805655 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-kube-api-access-lb8mz\" (UniqueName: \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-kube-api-access-lb8mz\") pod \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " Oct 30 22:10:15.806029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.805702 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istiod-ca-cert\") pod \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " Oct 30 22:10:15.806029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.805735 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-socket\") pod \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " Oct 30 22:10:15.806029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.805808 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-credential-socket\") pod \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " Oct 30 22:10:15.806327 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.806096 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-credential-socket" pod "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" (UID: "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.806327 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.806223 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istiod-ca-cert" pod "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" (UID: "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:15.806404 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.806332 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-data\") pod \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " Oct 30 22:10:15.806404 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.806383 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-certs\") pod \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " Oct 30 22:10:15.806483 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.806316 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-socket" pod "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" (UID: "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.806578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.806542 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-envoy\") pod \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\" (UID: \"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7\") " Oct 30 22:10:15.806578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.806557 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-data" pod "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" (UID: "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.806690 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.806658 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-certs" pod "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" (UID: "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.807331 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.807223 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.807331 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.807250 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.807331 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.807265 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.807331 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.807278 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.807331 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.807291 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.808276 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.808244 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-token" pod "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" (UID: "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:15.808938 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.808914 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-podinfo" pod "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" (UID: "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:15.809062 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.808996 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-envoy" pod "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" (UID: "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.809967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.809916 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-kube-api-access-lb8mz" pod "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" (UID: "f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7"). InnerVolumeSpecName "kube-api-access-lb8mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:15.907810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.907683 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9293cc9-9ba6-4328-a72d-df484d2388e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9293cc9-9ba6-4328-a72d-df484d2388e1-istiod-ca-cert\") pod \"e9293cc9-9ba6-4328-a72d-df484d2388e1\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " Oct 30 22:10:15.907810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.907730 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-token\" (UniqueName: \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-token\") pod \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " Oct 30 22:10:15.907810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.907755 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-kube-api-access-hh2h9\" (UniqueName: \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-kube-api-access-hh2h9\") pod \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " Oct 30 22:10:15.907810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.907800 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-socket\") pod \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " Oct 30 22:10:15.908196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.907844 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-credential-socket\") pod \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " Oct 30 22:10:15.908196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.907898 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-socket\") pod \"e9293cc9-9ba6-4328-a72d-df484d2388e1\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " Oct 30 22:10:15.908196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.907982 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-podinfo\") pod \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " Oct 30 22:10:15.908196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908009 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-token\" (UniqueName: \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-token\") pod \"e9293cc9-9ba6-4328-a72d-df484d2388e1\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " Oct 30 22:10:15.908196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908078 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-certs\") pod \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " Oct 30 22:10:15.908196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908109 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-credential-socket\") pod \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " Oct 30 22:10:15.908196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908147 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9293cc9-9ba6-4328-a72d-df484d2388e1-istiod-ca-cert" pod "e9293cc9-9ba6-4328-a72d-df484d2388e1" (UID: "e9293cc9-9ba6-4328-a72d-df484d2388e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:15.908196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908172 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-kube-api-access-8p88s\" (UniqueName: \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-kube-api-access-8p88s\") pod \"e9293cc9-9ba6-4328-a72d-df484d2388e1\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " Oct 30 22:10:15.908584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908219 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-certs\") pod \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " Oct 30 22:10:15.908584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908258 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-data\") pod \"e9293cc9-9ba6-4328-a72d-df484d2388e1\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " Oct 30 22:10:15.908584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908290 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-envoy\") pod \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " Oct 30 22:10:15.908584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908323 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-podinfo\") pod \"e9293cc9-9ba6-4328-a72d-df484d2388e1\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " Oct 30 22:10:15.908584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908353 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-socket\") pod \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " Oct 30 22:10:15.908584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908404 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-data\") pod \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " Oct 30 22:10:15.908584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908447 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-kube-api-access-96tgz\" (UniqueName: \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-kube-api-access-96tgz\") pod \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " Oct 30 22:10:15.908584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908477 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istiod-ca-cert\") pod \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " Oct 30 22:10:15.908584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908507 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-data\") pod \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " Oct 30 22:10:15.908584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908559 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istiod-ca-cert\") pod \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\" (UID: \"0630b556-6459-480b-abf9-a3f3d0ecc6c1\") " Oct 30 22:10:15.909080 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908639 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-certs\") pod \"e9293cc9-9ba6-4328-a72d-df484d2388e1\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " Oct 30 22:10:15.909080 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908672 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-envoy\") pod \"e9293cc9-9ba6-4328-a72d-df484d2388e1\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " Oct 30 22:10:15.909080 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908713 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-credential-socket\") pod \"e9293cc9-9ba6-4328-a72d-df484d2388e1\" (UID: \"e9293cc9-9ba6-4328-a72d-df484d2388e1\") " Oct 30 22:10:15.909080 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908767 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-token\" (UniqueName: \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-token\") pod \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " Oct 30 22:10:15.909080 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908807 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-envoy\") pod \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " Oct 30 22:10:15.909080 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908876 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-podinfo\") pod \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\" (UID: \"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e\") " Oct 30 22:10:15.909080 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.908878 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-certs" pod "0630b556-6459-480b-abf9-a3f3d0ecc6c1" (UID: "0630b556-6459-480b-abf9-a3f3d0ecc6c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.909399 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.909214 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-data" pod "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" (UID: "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.909876 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.909850 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-credential-socket" pod "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" (UID: "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.910015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.909883 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.910015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.909905 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.910015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.909921 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.910015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.909963 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.910015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.909978 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9293cc9-9ba6-4328-a72d-df484d2388e1-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.910015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.909992 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.910015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.910006 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb8mz\" (UniqueName: \"kubernetes.io/projected/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7-kube-api-access-lb8mz\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.910389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.910086 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-socket" pod "e9293cc9-9ba6-4328-a72d-df484d2388e1" (UID: "e9293cc9-9ba6-4328-a72d-df484d2388e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.910389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.910294 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-certs" pod "e9293cc9-9ba6-4328-a72d-df484d2388e1" (UID: "e9293cc9-9ba6-4328-a72d-df484d2388e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.910508 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.910484 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-socket" pod "0630b556-6459-480b-abf9-a3f3d0ecc6c1" (UID: "0630b556-6459-480b-abf9-a3f3d0ecc6c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.911447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.910800 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-data" pod "0630b556-6459-480b-abf9-a3f3d0ecc6c1" (UID: "0630b556-6459-480b-abf9-a3f3d0ecc6c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.911447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.910870 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-socket" pod "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" (UID: "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.911447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.910931 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-data" pod "e9293cc9-9ba6-4328-a72d-df484d2388e1" (UID: "e9293cc9-9ba6-4328-a72d-df484d2388e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.911447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.911197 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-credential-socket" pod "0630b556-6459-480b-abf9-a3f3d0ecc6c1" (UID: "0630b556-6459-480b-abf9-a3f3d0ecc6c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.911447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.911108 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istiod-ca-cert" pod "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" (UID: "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:15.911447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.911296 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istiod-ca-cert" pod "0630b556-6459-480b-abf9-a3f3d0ecc6c1" (UID: "0630b556-6459-480b-abf9-a3f3d0ecc6c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:15.911447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.911402 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-certs" pod "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" (UID: "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.911447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.911418 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-credential-socket" pod "e9293cc9-9ba6-4328-a72d-df484d2388e1" (UID: "e9293cc9-9ba6-4328-a72d-df484d2388e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.914303 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.914276 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-podinfo" pod "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" (UID: "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:15.914867 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.914840 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-kube-api-access-8p88s" pod "e9293cc9-9ba6-4328-a72d-df484d2388e1" (UID: "e9293cc9-9ba6-4328-a72d-df484d2388e1"). InnerVolumeSpecName "kube-api-access-8p88s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:15.916002 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.915934 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-kube-api-access-96tgz" pod "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" (UID: "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e"). InnerVolumeSpecName "kube-api-access-96tgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:15.916002 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.915992 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-podinfo" pod "e9293cc9-9ba6-4328-a72d-df484d2388e1" (UID: "e9293cc9-9ba6-4328-a72d-df484d2388e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:15.916123 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.916020 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-token" pod "e9293cc9-9ba6-4328-a72d-df484d2388e1" (UID: "e9293cc9-9ba6-4328-a72d-df484d2388e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:15.916123 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.916097 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-kube-api-access-hh2h9" pod "0630b556-6459-480b-abf9-a3f3d0ecc6c1" (UID: "0630b556-6459-480b-abf9-a3f3d0ecc6c1"). InnerVolumeSpecName "kube-api-access-hh2h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:15.916231 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.916181 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-token" pod "0630b556-6459-480b-abf9-a3f3d0ecc6c1" (UID: "0630b556-6459-480b-abf9-a3f3d0ecc6c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:15.916285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.916250 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-podinfo" pod "0630b556-6459-480b-abf9-a3f3d0ecc6c1" (UID: "0630b556-6459-480b-abf9-a3f3d0ecc6c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:15.916558 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.916527 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-token" pod "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" (UID: "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:15.916731 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.916704 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-envoy" pod "0630b556-6459-480b-abf9-a3f3d0ecc6c1" (UID: "0630b556-6459-480b-abf9-a3f3d0ecc6c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.916788 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.916766 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-envoy" pod "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" (UID: "8edd5bc5-174f-4554-9d7d-a5f0d9e1170e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:15.917889 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:15.917870 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-envoy" pod "e9293cc9-9ba6-4328-a72d-df484d2388e1" (UID: "e9293cc9-9ba6-4328-a72d-df484d2388e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:16.022784 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022740 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.022784 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022785 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022804 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022818 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022832 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022846 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022863 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p88s\" (UniqueName: \"kubernetes.io/projected/e9293cc9-9ba6-4328-a72d-df484d2388e1-kube-api-access-8p88s\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022876 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022889 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022901 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022915 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022927 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96tgz\" (UniqueName: \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-kube-api-access-96tgz\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022954 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022968 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022981 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.022993 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.023006 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.023020 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293cc9-9ba6-4328-a72d-df484d2388e1-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.023035 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.023048 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.023060 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.023073 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.023086 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh2h9\" (UniqueName: \"kubernetes.io/projected/0630b556-6459-480b-abf9-a3f3d0ecc6c1-kube-api-access-hh2h9\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.029434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.023100 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.041578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.041543 4859 generic.go:356] "Generic (PLEG): container finished" podID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerID="a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9" exitCode=0 Oct 30 22:10:16.041769 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.041599 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" Oct 30 22:10:16.041769 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.041745 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" event={"ID":"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e","Type":"ContainerDied","Data":"a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9"} Oct 30 22:10:16.041892 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.041786 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57" event={"ID":"8edd5bc5-174f-4554-9d7d-a5f0d9e1170e","Type":"ContainerDied","Data":"5205d4a4def91d44a75221d546f40c809a796823b6923613f494bb02d5c266b1"} Oct 30 22:10:16.041892 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.041812 4859 scope.go:122] "RemoveContainer" containerID="a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9" Oct 30 22:10:16.046069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.045385 4859 generic.go:356] "Generic (PLEG): container finished" podID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerID="2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267" exitCode=0 Oct 30 22:10:16.046069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.045500 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" event={"ID":"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7","Type":"ContainerDied","Data":"2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267"} Oct 30 22:10:16.046069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.045553 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" event={"ID":"f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7","Type":"ContainerDied","Data":"741df6166069c03b8111284f8db629f3581dc98ae2083b1327d010ed5d2a79fa"} Oct 30 22:10:16.046069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.045689 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k" Oct 30 22:10:16.053445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.052570 4859 generic.go:356] "Generic (PLEG): container finished" podID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerID="8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a" exitCode=0 Oct 30 22:10:16.053445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.052674 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" Oct 30 22:10:16.053445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.052771 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" event={"ID":"0630b556-6459-480b-abf9-a3f3d0ecc6c1","Type":"ContainerDied","Data":"8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a"} Oct 30 22:10:16.053445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.052811 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p" event={"ID":"0630b556-6459-480b-abf9-a3f3d0ecc6c1","Type":"ContainerDied","Data":"d7801f62f4c8c4a988773267985e7f45d050263eda90b32822a28c574248109b"} Oct 30 22:10:16.057976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.055404 4859 generic.go:356] "Generic (PLEG): container finished" podID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerID="6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3" exitCode=0 Oct 30 22:10:16.057976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.055454 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" event={"ID":"e9293cc9-9ba6-4328-a72d-df484d2388e1","Type":"ContainerDied","Data":"6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3"} Oct 30 22:10:16.057976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.055502 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" event={"ID":"e9293cc9-9ba6-4328-a72d-df484d2388e1","Type":"ContainerDied","Data":"72c0bb5124604447aae2843af74df6db0e28d2f325f014ce9d19e1e164152c2d"} Oct 30 22:10:16.057976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.055582 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt" Oct 30 22:10:16.078777 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.078753 4859 scope.go:122] "RemoveContainer" containerID="a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9" Oct 30 22:10:16.079277 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:10:16.079245 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9\": container with ID starting with a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9 not found: ID does not exist" containerID="a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9" Oct 30 22:10:16.079380 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.079290 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9"} err="failed to get container status \"a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9\": rpc error: code = NotFound desc = could not find container \"a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9\": container with ID starting with a7ccd2721b49487193d7c99798f5cd6106d31374097386d606d44c7a1631a8a9 not found: ID does not exist" Oct 30 22:10:16.079380 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.079318 4859 scope.go:122] "RemoveContainer" containerID="2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267" Oct 30 22:10:16.094158 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.094137 4859 scope.go:122] "RemoveContainer" containerID="22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c" Oct 30 22:10:16.104760 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.104734 4859 scope.go:122] "RemoveContainer" containerID="2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267" Oct 30 22:10:16.105163 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:10:16.105138 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267\": container with ID starting with 2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267 not found: ID does not exist" containerID="2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267" Oct 30 22:10:16.105270 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.105170 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267"} err="failed to get container status \"2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267\": rpc error: code = NotFound desc = could not find container \"2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267\": container with ID starting with 2b4f890b91860245ff954fcdda699fda8f3d7e69380b441a038e45eecafdc267 not found: ID does not exist" Oct 30 22:10:16.105270 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.105188 4859 scope.go:122] "RemoveContainer" containerID="22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c" Oct 30 22:10:16.105638 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:10:16.105602 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c\": container with ID starting with 22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c not found: ID does not exist" containerID="22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c" Oct 30 22:10:16.105739 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.105645 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c"} err="failed to get container status \"22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c\": rpc error: code = NotFound desc = could not find container \"22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c\": container with ID starting with 22e2f196e67c336f5c4f9fda9e3013d0291022e4e567363a378f3992df75a82c not found: ID does not exist" Oct 30 22:10:16.105739 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.105661 4859 scope.go:122] "RemoveContainer" containerID="8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a" Oct 30 22:10:16.120454 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.120432 4859 scope.go:122] "RemoveContainer" containerID="8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a" Oct 30 22:10:16.120871 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:10:16.120851 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a\": container with ID starting with 8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a not found: ID does not exist" containerID="8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a" Oct 30 22:10:16.120919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.120883 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a"} err="failed to get container status \"8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a\": rpc error: code = NotFound desc = could not find container \"8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a\": container with ID starting with 8d66a0c75fd4ec5dc78168ba99a59618b93fdfee4e7d610feaf9ccf95e523c7a not found: ID does not exist" Oct 30 22:10:16.120919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.120902 4859 scope.go:122] "RemoveContainer" containerID="6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3" Oct 30 22:10:16.135145 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.135122 4859 scope.go:122] "RemoveContainer" containerID="6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3" Oct 30 22:10:16.135536 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:10:16.135499 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3\": container with ID starting with 6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3 not found: ID does not exist" containerID="6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3" Oct 30 22:10:16.135659 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.135542 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3"} err="failed to get container status \"6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3\": rpc error: code = NotFound desc = could not find container \"6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3\": container with ID starting with 6a988e1c586999d69b792826ea945bab176edc3c1a4e045a5406f975a576d7d3 not found: ID does not exist" Oct 30 22:10:16.352404 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.352370 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57"] Oct 30 22:10:16.356680 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.356648 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-77k57"] Oct 30 22:10:16.463045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.463011 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p"] Oct 30 22:10:16.469293 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.469249 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hdn5p"] Oct 30 22:10:16.575248 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.575207 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt"] Oct 30 22:10:16.581167 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.581134 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-428wt"] Oct 30 22:10:16.692986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.692933 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k"] Oct 30 22:10:16.697722 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.697684 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-kpb9k"] Oct 30 22:10:16.742349 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.742316 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" path="/var/lib/kubelet/pods/0630b556-6459-480b-abf9-a3f3d0ecc6c1/volumes" Oct 30 22:10:16.742840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.742825 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" path="/var/lib/kubelet/pods/8edd5bc5-174f-4554-9d7d-a5f0d9e1170e/volumes" Oct 30 22:10:16.743294 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.743280 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" path="/var/lib/kubelet/pods/e9293cc9-9ba6-4328-a72d-df484d2388e1/volumes" Oct 30 22:10:16.744031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:16.744013 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" path="/var/lib/kubelet/pods/f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7/volumes" Oct 30 22:10:27.965874 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:27.965832 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz"] Oct 30 22:10:27.966393 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:27.966060 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" containerID="cri-o://60190fad34a88d979de72d2526ca01b451df973b02571109bd0ed987c69ce245" gracePeriod=30 Oct 30 22:10:28.014247 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:28.014209 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8"] Oct 30 22:10:28.014446 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:28.014426 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" containerID="cri-o://07aa472f1aa2bea104b6a09c5bf01d45886ed86ec98a64f9de77a6d0b2565a26" gracePeriod=30 Oct 30 22:10:28.017971 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:28.017924 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw"] Oct 30 22:10:28.018171 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:28.018151 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" containerID="cri-o://c5cd7522cfb3a2e1d7e16c9e94ab944a33ea0f36aa85953d8286fd115acfcbed" gracePeriod=30 Oct 30 22:10:28.063535 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:28.063502 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772"] Oct 30 22:10:28.063708 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:28.063687 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" containerID="cri-o://27b804637390e175875e42aff7954a86578a67c7507399e8ff1e579fb407e195" gracePeriod=30 Oct 30 22:10:30.982177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:30.982135 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:10:30.982641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:30.982203 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:10:33.093184 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.093153 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:10:33.093543 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.093211 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:10:33.229748 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.229490 4859 generic.go:356] "Generic (PLEG): container finished" podID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerID="60190fad34a88d979de72d2526ca01b451df973b02571109bd0ed987c69ce245" exitCode=0 Oct 30 22:10:33.229748 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.229665 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" event={"ID":"b698de93-8da9-45f6-954f-c8d7c64118d3","Type":"ContainerDied","Data":"60190fad34a88d979de72d2526ca01b451df973b02571109bd0ed987c69ce245"} Oct 30 22:10:33.237189 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.236936 4859 generic.go:356] "Generic (PLEG): container finished" podID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerID="07aa472f1aa2bea104b6a09c5bf01d45886ed86ec98a64f9de77a6d0b2565a26" exitCode=0 Oct 30 22:10:33.237189 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.237061 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" event={"ID":"784a3a6f-5419-48d6-a7da-3eb93c205b9b","Type":"ContainerDied","Data":"07aa472f1aa2bea104b6a09c5bf01d45886ed86ec98a64f9de77a6d0b2565a26"} Oct 30 22:10:33.239583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.239372 4859 generic.go:356] "Generic (PLEG): container finished" podID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerID="c5cd7522cfb3a2e1d7e16c9e94ab944a33ea0f36aa85953d8286fd115acfcbed" exitCode=0 Oct 30 22:10:33.239583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.239518 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" event={"ID":"748bacd4-dd89-4c59-8be2-1a8bae565bb7","Type":"ContainerDied","Data":"c5cd7522cfb3a2e1d7e16c9e94ab944a33ea0f36aa85953d8286fd115acfcbed"} Oct 30 22:10:33.241933 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.241779 4859 generic.go:356] "Generic (PLEG): container finished" podID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerID="27b804637390e175875e42aff7954a86578a67c7507399e8ff1e579fb407e195" exitCode=0 Oct 30 22:10:33.241933 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.241874 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" event={"ID":"92deb8c0-dc69-4cba-af2a-a77ed60be8cc","Type":"ContainerDied","Data":"27b804637390e175875e42aff7954a86578a67c7507399e8ff1e579fb407e195"} Oct 30 22:10:33.335043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.334768 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:10:33.423915 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.423827 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:10:33.477714 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.477482 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-data\") pod \"b698de93-8da9-45f6-954f-c8d7c64118d3\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " Oct 30 22:10:33.477861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.477684 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-data" pod "b698de93-8da9-45f6-954f-c8d7c64118d3" (UID: "b698de93-8da9-45f6-954f-c8d7c64118d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.477929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.477865 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-credential-socket\") pod \"b698de93-8da9-45f6-954f-c8d7c64118d3\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " Oct 30 22:10:33.478114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.478094 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-credential-socket" pod "b698de93-8da9-45f6-954f-c8d7c64118d3" (UID: "b698de93-8da9-45f6-954f-c8d7c64118d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.478216 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.478201 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-certs\") pod \"b698de93-8da9-45f6-954f-c8d7c64118d3\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " Oct 30 22:10:33.478455 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.478436 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-certs" pod "b698de93-8da9-45f6-954f-c8d7c64118d3" (UID: "b698de93-8da9-45f6-954f-c8d7c64118d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.478542 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.478496 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-kube-api-access-rr2ch\" (UniqueName: \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-kube-api-access-rr2ch\") pod \"b698de93-8da9-45f6-954f-c8d7c64118d3\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " Oct 30 22:10:33.480048 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.480029 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-token\" (UniqueName: \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-token\") pod \"b698de93-8da9-45f6-954f-c8d7c64118d3\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " Oct 30 22:10:33.480236 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.480215 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-envoy\") pod \"b698de93-8da9-45f6-954f-c8d7c64118d3\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " Oct 30 22:10:33.480392 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.480376 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-socket\") pod \"b698de93-8da9-45f6-954f-c8d7c64118d3\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " Oct 30 22:10:33.480508 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.480493 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-podinfo\") pod \"b698de93-8da9-45f6-954f-c8d7c64118d3\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " Oct 30 22:10:33.480642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.480629 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b698de93-8da9-45f6-954f-c8d7c64118d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b698de93-8da9-45f6-954f-c8d7c64118d3-istiod-ca-cert\") pod \"b698de93-8da9-45f6-954f-c8d7c64118d3\" (UID: \"b698de93-8da9-45f6-954f-c8d7c64118d3\") " Oct 30 22:10:33.481528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.481094 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-kube-api-access-rr2ch" pod "b698de93-8da9-45f6-954f-c8d7c64118d3" (UID: "b698de93-8da9-45f6-954f-c8d7c64118d3"). InnerVolumeSpecName "kube-api-access-rr2ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:33.481528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.481116 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-socket" pod "b698de93-8da9-45f6-954f-c8d7c64118d3" (UID: "b698de93-8da9-45f6-954f-c8d7c64118d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.481528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.481384 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b698de93-8da9-45f6-954f-c8d7c64118d3-istiod-ca-cert" pod "b698de93-8da9-45f6-954f-c8d7c64118d3" (UID: "b698de93-8da9-45f6-954f-c8d7c64118d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:33.481528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.481433 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.481528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.481452 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.481528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.481466 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr2ch\" (UniqueName: \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-kube-api-access-rr2ch\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.481528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.481483 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.481528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.481496 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.483135 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.482488 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-token" pod "b698de93-8da9-45f6-954f-c8d7c64118d3" (UID: "b698de93-8da9-45f6-954f-c8d7c64118d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:33.483135 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.482817 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-envoy" pod "b698de93-8da9-45f6-954f-c8d7c64118d3" (UID: "b698de93-8da9-45f6-954f-c8d7c64118d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.484242 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.484215 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-podinfo" pod "b698de93-8da9-45f6-954f-c8d7c64118d3" (UID: "b698de93-8da9-45f6-954f-c8d7c64118d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:33.514783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.514726 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 22:10:33.517454 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.517432 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:10:33.583295 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.583257 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-credential-socket\") pod \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " Oct 30 22:10:33.583739 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.583715 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-certs\") pod \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " Oct 30 22:10:33.583912 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.583898 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-data\") pod \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " Oct 30 22:10:33.584042 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.584027 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-token\" (UniqueName: \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-token\") pod \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " Oct 30 22:10:33.584182 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.584168 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-kube-api-access-dqdkl\" (UniqueName: \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-kube-api-access-dqdkl\") pod \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " Oct 30 22:10:33.584292 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.584279 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-podinfo\") pod \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " Oct 30 22:10:33.584479 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.584405 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-envoy\") pod \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " Oct 30 22:10:33.584606 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.584527 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-socket\") pod \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " Oct 30 22:10:33.584692 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.584676 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istiod-ca-cert\") pod \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\" (UID: \"748bacd4-dd89-4c59-8be2-1a8bae565bb7\") " Oct 30 22:10:33.588925 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.588570 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-certs" pod "748bacd4-dd89-4c59-8be2-1a8bae565bb7" (UID: "748bacd4-dd89-4c59-8be2-1a8bae565bb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.588925 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.588756 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-credential-socket" pod "748bacd4-dd89-4c59-8be2-1a8bae565bb7" (UID: "748bacd4-dd89-4c59-8be2-1a8bae565bb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.589454 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.589369 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-data" pod "748bacd4-dd89-4c59-8be2-1a8bae565bb7" (UID: "748bacd4-dd89-4c59-8be2-1a8bae565bb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.593347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.590477 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-socket" pod "748bacd4-dd89-4c59-8be2-1a8bae565bb7" (UID: "748bacd4-dd89-4c59-8be2-1a8bae565bb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.593347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.593149 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-kube-api-access-dqdkl" pod "748bacd4-dd89-4c59-8be2-1a8bae565bb7" (UID: "748bacd4-dd89-4c59-8be2-1a8bae565bb7"). InnerVolumeSpecName "kube-api-access-dqdkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:33.593347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.593225 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.593347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.593247 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.593347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.593262 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.593347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.593277 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.593347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.593289 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.593347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.593304 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b698de93-8da9-45f6-954f-c8d7c64118d3-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.593347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.593318 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b698de93-8da9-45f6-954f-c8d7c64118d3-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.593347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.593331 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.596477 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.596436 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-envoy" pod "748bacd4-dd89-4c59-8be2-1a8bae565bb7" (UID: "748bacd4-dd89-4c59-8be2-1a8bae565bb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.597822 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.597790 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-token" pod "748bacd4-dd89-4c59-8be2-1a8bae565bb7" (UID: "748bacd4-dd89-4c59-8be2-1a8bae565bb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:33.600424 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.600039 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-podinfo" pod "748bacd4-dd89-4c59-8be2-1a8bae565bb7" (UID: "748bacd4-dd89-4c59-8be2-1a8bae565bb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:33.602575 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.602526 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istiod-ca-cert" pod "748bacd4-dd89-4c59-8be2-1a8bae565bb7" (UID: "748bacd4-dd89-4c59-8be2-1a8bae565bb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:33.694269 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694230 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istiod-ca-cert\") pod \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " Oct 30 22:10:33.694420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694296 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-credential-socket\") pod \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " Oct 30 22:10:33.694420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694333 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-token\" (UniqueName: \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-token\") pod \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " Oct 30 22:10:33.694420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694375 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-socket\") pod \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " Oct 30 22:10:33.694560 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694430 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-certs\") pod \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " Oct 30 22:10:33.694560 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694512 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-envoy\") pod \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " Oct 30 22:10:33.694675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694562 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-podinfo\") pod \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " Oct 30 22:10:33.694675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694595 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istiod-ca-cert\") pod \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " Oct 30 22:10:33.694675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694633 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-data\") pod \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " Oct 30 22:10:33.694675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694642 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-credential-socket" pod "92deb8c0-dc69-4cba-af2a-a77ed60be8cc" (UID: "92deb8c0-dc69-4cba-af2a-a77ed60be8cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.694675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694663 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-kube-api-access-zdk6t\" (UniqueName: \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-kube-api-access-zdk6t\") pod \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " Oct 30 22:10:33.694675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694651 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-socket" pod "784a3a6f-5419-48d6-a7da-3eb93c205b9b" (UID: "784a3a6f-5419-48d6-a7da-3eb93c205b9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.695021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694696 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istiod-ca-cert" pod "92deb8c0-dc69-4cba-af2a-a77ed60be8cc" (UID: "92deb8c0-dc69-4cba-af2a-a77ed60be8cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:33.695021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694792 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-certs" pod "784a3a6f-5419-48d6-a7da-3eb93c205b9b" (UID: "784a3a6f-5419-48d6-a7da-3eb93c205b9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.695021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694868 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-data" pod "784a3a6f-5419-48d6-a7da-3eb93c205b9b" (UID: "784a3a6f-5419-48d6-a7da-3eb93c205b9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.695021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.694981 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istiod-ca-cert" pod "784a3a6f-5419-48d6-a7da-3eb93c205b9b" (UID: "784a3a6f-5419-48d6-a7da-3eb93c205b9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:33.695705 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.695677 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-token\" (UniqueName: \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-token\") pod \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " Oct 30 22:10:33.695824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.695721 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-data\") pod \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " Oct 30 22:10:33.695824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.695766 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-socket\") pod \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " Oct 30 22:10:33.695824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.695795 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-envoy\") pod \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " Oct 30 22:10:33.696117 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.695846 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-podinfo\") pod \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " Oct 30 22:10:33.696117 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.695939 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-certs\") pod \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " Oct 30 22:10:33.696117 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696089 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-kube-api-access-gxg2l\" (UniqueName: \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-kube-api-access-gxg2l\") pod \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\" (UID: \"92deb8c0-dc69-4cba-af2a-a77ed60be8cc\") " Oct 30 22:10:33.696271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696125 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-credential-socket\") pod \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\" (UID: \"784a3a6f-5419-48d6-a7da-3eb93c205b9b\") " Oct 30 22:10:33.696271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696193 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-socket" pod "92deb8c0-dc69-4cba-af2a-a77ed60be8cc" (UID: "92deb8c0-dc69-4cba-af2a-a77ed60be8cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.696411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696388 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-data" pod "92deb8c0-dc69-4cba-af2a-a77ed60be8cc" (UID: "92deb8c0-dc69-4cba-af2a-a77ed60be8cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.696629 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696607 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-credential-socket" pod "784a3a6f-5419-48d6-a7da-3eb93c205b9b" (UID: "784a3a6f-5419-48d6-a7da-3eb93c205b9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.696888 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696862 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696894 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696910 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696925 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696956 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696971 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696984 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.696997 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697011 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697024 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697038 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqdkl\" (UniqueName: \"kubernetes.io/projected/748bacd4-dd89-4c59-8be2-1a8bae565bb7-kube-api-access-dqdkl\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697052 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697066 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748bacd4-dd89-4c59-8be2-1a8bae565bb7-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697080 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.697466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697278 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-envoy" pod "92deb8c0-dc69-4cba-af2a-a77ed60be8cc" (UID: "92deb8c0-dc69-4cba-af2a-a77ed60be8cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.697466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697292 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-certs" pod "92deb8c0-dc69-4cba-af2a-a77ed60be8cc" (UID: "92deb8c0-dc69-4cba-af2a-a77ed60be8cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.697466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697318 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-podinfo" pod "784a3a6f-5419-48d6-a7da-3eb93c205b9b" (UID: "784a3a6f-5419-48d6-a7da-3eb93c205b9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:33.697466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697366 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-token" pod "92deb8c0-dc69-4cba-af2a-a77ed60be8cc" (UID: "92deb8c0-dc69-4cba-af2a-a77ed60be8cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:33.697880 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.697852 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-kube-api-access-zdk6t" pod "784a3a6f-5419-48d6-a7da-3eb93c205b9b" (UID: "784a3a6f-5419-48d6-a7da-3eb93c205b9b"). InnerVolumeSpecName "kube-api-access-zdk6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:33.699759 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.699735 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-envoy" pod "784a3a6f-5419-48d6-a7da-3eb93c205b9b" (UID: "784a3a6f-5419-48d6-a7da-3eb93c205b9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:33.700328 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.700223 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-podinfo" pod "92deb8c0-dc69-4cba-af2a-a77ed60be8cc" (UID: "92deb8c0-dc69-4cba-af2a-a77ed60be8cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:33.701199 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.701172 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-kube-api-access-gxg2l" pod "92deb8c0-dc69-4cba-af2a-a77ed60be8cc" (UID: "92deb8c0-dc69-4cba-af2a-a77ed60be8cc"). InnerVolumeSpecName "kube-api-access-gxg2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:33.702785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.702760 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-token" pod "784a3a6f-5419-48d6-a7da-3eb93c205b9b" (UID: "784a3a6f-5419-48d6-a7da-3eb93c205b9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:33.798673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.798597 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.798673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.798625 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdk6t\" (UniqueName: \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-kube-api-access-zdk6t\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.798673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.798635 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.798673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.798643 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784a3a6f-5419-48d6-a7da-3eb93c205b9b-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.798673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.798653 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.798673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.798660 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.798673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.798669 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxg2l\" (UniqueName: \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-kube-api-access-gxg2l\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.798673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.798679 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.799066 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:33.798687 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92deb8c0-dc69-4cba-af2a-a77ed60be8cc-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:34.251571 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.251538 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" Oct 30 22:10:34.252031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.251543 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz" event={"ID":"b698de93-8da9-45f6-954f-c8d7c64118d3","Type":"ContainerDied","Data":"6e6e12ccf1bf98850eb28da57a0a77dc45ef85083fd8de992eadc1d6cd6c7858"} Oct 30 22:10:34.252031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.251689 4859 scope.go:122] "RemoveContainer" containerID="60190fad34a88d979de72d2526ca01b451df973b02571109bd0ed987c69ce245" Oct 30 22:10:34.253107 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.253080 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" event={"ID":"784a3a6f-5419-48d6-a7da-3eb93c205b9b","Type":"ContainerDied","Data":"503283aaff860097a6ee354faf5f665c510d86bf31768b2221206527ae7237d9"} Oct 30 22:10:34.253181 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.253088 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8" Oct 30 22:10:34.254741 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.254721 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" Oct 30 22:10:34.255292 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.254768 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw" event={"ID":"748bacd4-dd89-4c59-8be2-1a8bae565bb7","Type":"ContainerDied","Data":"85c3e712c681a8cc61b2841aa5a76661a7ef6ed8bd86c658109d41867e86ece8"} Oct 30 22:10:34.264285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.264187 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" Oct 30 22:10:34.264285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.264180 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772" event={"ID":"92deb8c0-dc69-4cba-af2a-a77ed60be8cc","Type":"ContainerDied","Data":"314b3072f27fac1d574efcb12cd5f905aaba267e23866f701d1b3d338aa3b1a5"} Oct 30 22:10:34.268486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.268470 4859 scope.go:122] "RemoveContainer" containerID="07aa472f1aa2bea104b6a09c5bf01d45886ed86ec98a64f9de77a6d0b2565a26" Oct 30 22:10:34.280331 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.280310 4859 scope.go:122] "RemoveContainer" containerID="c5cd7522cfb3a2e1d7e16c9e94ab944a33ea0f36aa85953d8286fd115acfcbed" Oct 30 22:10:34.293727 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.293382 4859 scope.go:122] "RemoveContainer" containerID="27b804637390e175875e42aff7954a86578a67c7507399e8ff1e579fb407e195" Oct 30 22:10:34.417879 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.417840 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz"] Oct 30 22:10:34.421280 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.421254 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-qw9sz"] Oct 30 22:10:34.550052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.548400 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8"] Oct 30 22:10:34.555988 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.554794 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-nx6b8"] Oct 30 22:10:34.660118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.660082 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772"] Oct 30 22:10:34.666011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.665977 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-59772"] Oct 30 22:10:34.742974 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.742929 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" path="/var/lib/kubelet/pods/784a3a6f-5419-48d6-a7da-3eb93c205b9b/volumes" Oct 30 22:10:34.743457 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.743443 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" path="/var/lib/kubelet/pods/92deb8c0-dc69-4cba-af2a-a77ed60be8cc/volumes" Oct 30 22:10:34.743864 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.743848 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" path="/var/lib/kubelet/pods/b698de93-8da9-45f6-954f-c8d7c64118d3/volumes" Oct 30 22:10:34.778367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.778330 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw"] Oct 30 22:10:34.781911 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:34.781885 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-d28jw"] Oct 30 22:10:36.743060 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:10:36.743022 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" path="/var/lib/kubelet/pods/748bacd4-dd89-4c59-8be2-1a8bae565bb7/volumes" Oct 30 22:10:39.881442 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:10:39.881406 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:11:04.023759 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:04.023708 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:11:09.895989 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:11:09.895785 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:11:10.562560 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:10.562526 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p"] Oct 30 22:11:10.562756 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:10.562731 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerName="istio-proxy" containerID="cri-o://02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b" gracePeriod=30 Oct 30 22:11:10.612374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:10.612331 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd"] Oct 30 22:11:10.612574 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:10.612550 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" containerID="cri-o://dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c" gracePeriod=30 Oct 30 22:11:14.948145 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:14.948105 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:11:14.948551 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:14.948171 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:11:15.033094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:15.033056 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:11:15.033255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:15.033125 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:11:16.054309 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.054289 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:11:16.057401 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.057382 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:11:16.146878 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.146849 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-token\" (UniqueName: \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-token\") pod \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " Oct 30 22:11:16.147067 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.146895 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-kube-api-access-xx5fb\" (UniqueName: \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-kube-api-access-xx5fb\") pod \"0dec6bd6-145b-4257-8077-4588a7e68960\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " Oct 30 22:11:16.147067 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.146923 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-credential-socket\") pod \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " Oct 30 22:11:16.147067 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147044 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-envoy\") pod \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " Oct 30 22:11:16.147262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147085 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-kube-api-access-vll9j\" (UniqueName: \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-kube-api-access-vll9j\") pod \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " Oct 30 22:11:16.147262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147121 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dec6bd6-145b-4257-8077-4588a7e68960-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dec6bd6-145b-4257-8077-4588a7e68960-istiod-ca-cert\") pod \"0dec6bd6-145b-4257-8077-4588a7e68960\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " Oct 30 22:11:16.147262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147159 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-podinfo\") pod \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " Oct 30 22:11:16.147262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147207 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-credential-socket" pod "3ed1f318-6c24-4904-8a3d-64a71fcab80e" (UID: "3ed1f318-6c24-4904-8a3d-64a71fcab80e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:16.147262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147238 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-data\") pod \"0dec6bd6-145b-4257-8077-4588a7e68960\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " Oct 30 22:11:16.147559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147363 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-certs\") pod \"0dec6bd6-145b-4257-8077-4588a7e68960\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " Oct 30 22:11:16.147559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147456 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-certs\") pod \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " Oct 30 22:11:16.147559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147506 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-credential-socket\") pod \"0dec6bd6-145b-4257-8077-4588a7e68960\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " Oct 30 22:11:16.147559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147540 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-socket\") pod \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " Oct 30 22:11:16.147765 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147588 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-data" pod "0dec6bd6-145b-4257-8077-4588a7e68960" (UID: "0dec6bd6-145b-4257-8077-4588a7e68960"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:16.147765 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147592 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dec6bd6-145b-4257-8077-4588a7e68960-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dec6bd6-145b-4257-8077-4588a7e68960-istio-podinfo\") pod \"0dec6bd6-145b-4257-8077-4588a7e68960\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " Oct 30 22:11:16.147765 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147679 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-socket\") pod \"0dec6bd6-145b-4257-8077-4588a7e68960\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " Oct 30 22:11:16.147765 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147712 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-data\") pod \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " Oct 30 22:11:16.148012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147765 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istiod-ca-cert\") pod \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\" (UID: \"3ed1f318-6c24-4904-8a3d-64a71fcab80e\") " Oct 30 22:11:16.148012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147833 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-istio-token\" (UniqueName: \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-istio-token\") pod \"0dec6bd6-145b-4257-8077-4588a7e68960\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " Oct 30 22:11:16.148012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.147864 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-envoy\") pod \"0dec6bd6-145b-4257-8077-4588a7e68960\" (UID: \"0dec6bd6-145b-4257-8077-4588a7e68960\") " Oct 30 22:11:16.148489 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.148464 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.148577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.148495 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.148889 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dec6bd6-145b-4257-8077-4588a7e68960-istiod-ca-cert" pod "0dec6bd6-145b-4257-8077-4588a7e68960" (UID: "0dec6bd6-145b-4257-8077-4588a7e68960"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149279 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-certs" pod "3ed1f318-6c24-4904-8a3d-64a71fcab80e" (UID: "3ed1f318-6c24-4904-8a3d-64a71fcab80e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149458 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-certs" pod "0dec6bd6-145b-4257-8077-4588a7e68960" (UID: "0dec6bd6-145b-4257-8077-4588a7e68960"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149536 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-kube-api-access-xx5fb" pod "0dec6bd6-145b-4257-8077-4588a7e68960" (UID: "0dec6bd6-145b-4257-8077-4588a7e68960"). InnerVolumeSpecName "kube-api-access-xx5fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149600 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-envoy" pod "3ed1f318-6c24-4904-8a3d-64a71fcab80e" (UID: "3ed1f318-6c24-4904-8a3d-64a71fcab80e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149608 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-token" pod "3ed1f318-6c24-4904-8a3d-64a71fcab80e" (UID: "3ed1f318-6c24-4904-8a3d-64a71fcab80e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149658 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-credential-socket" pod "0dec6bd6-145b-4257-8077-4588a7e68960" (UID: "0dec6bd6-145b-4257-8077-4588a7e68960"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149664 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-podinfo" pod "3ed1f318-6c24-4904-8a3d-64a71fcab80e" (UID: "3ed1f318-6c24-4904-8a3d-64a71fcab80e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149786 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-socket" pod "3ed1f318-6c24-4904-8a3d-64a71fcab80e" (UID: "3ed1f318-6c24-4904-8a3d-64a71fcab80e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149865 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-data" pod "3ed1f318-6c24-4904-8a3d-64a71fcab80e" (UID: "3ed1f318-6c24-4904-8a3d-64a71fcab80e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149886 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-kube-api-access-vll9j" pod "3ed1f318-6c24-4904-8a3d-64a71fcab80e" (UID: "3ed1f318-6c24-4904-8a3d-64a71fcab80e"). InnerVolumeSpecName "kube-api-access-vll9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:16.149981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.149925 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-socket" pod "0dec6bd6-145b-4257-8077-4588a7e68960" (UID: "0dec6bd6-145b-4257-8077-4588a7e68960"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:16.150464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.150015 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istiod-ca-cert" pod "3ed1f318-6c24-4904-8a3d-64a71fcab80e" (UID: "3ed1f318-6c24-4904-8a3d-64a71fcab80e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:16.150464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.150226 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dec6bd6-145b-4257-8077-4588a7e68960-istio-podinfo" pod "0dec6bd6-145b-4257-8077-4588a7e68960" (UID: "0dec6bd6-145b-4257-8077-4588a7e68960"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:16.150595 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.150576 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-envoy" pod "0dec6bd6-145b-4257-8077-4588a7e68960" (UID: "0dec6bd6-145b-4257-8077-4588a7e68960"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:16.151398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.151379 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-istio-token" pod "0dec6bd6-145b-4257-8077-4588a7e68960" (UID: "0dec6bd6-145b-4257-8077-4588a7e68960"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:16.249540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249461 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.249540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249495 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx5fb\" (UniqueName: \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-kube-api-access-xx5fb\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.249540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249506 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.249540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249514 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vll9j\" (UniqueName: \"kubernetes.io/projected/3ed1f318-6c24-4904-8a3d-64a71fcab80e-kube-api-access-vll9j\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.249540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249524 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dec6bd6-145b-4257-8077-4588a7e68960-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.249540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249532 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.249540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249541 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.250025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249550 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.250025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249559 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.250025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249567 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.250025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249575 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dec6bd6-145b-4257-8077-4588a7e68960-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.250025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249590 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.250025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249601 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.250025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249611 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed1f318-6c24-4904-8a3d-64a71fcab80e-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.250025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249619 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dec6bd6-145b-4257-8077-4588a7e68960-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.250025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.249627 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dec6bd6-145b-4257-8077-4588a7e68960-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:16.679343 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.679311 4859 generic.go:356] "Generic (PLEG): container finished" podID="0dec6bd6-145b-4257-8077-4588a7e68960" containerID="dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c" exitCode=0 Oct 30 22:11:16.679521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.679358 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" Oct 30 22:11:16.679521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.679397 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" event={"ID":"0dec6bd6-145b-4257-8077-4588a7e68960","Type":"ContainerDied","Data":"dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c"} Oct 30 22:11:16.679521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.679452 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd" event={"ID":"0dec6bd6-145b-4257-8077-4588a7e68960","Type":"ContainerDied","Data":"64312dba91f55e43f36dc9450e70631c174f1cde22ed755a396cb1aa23852fac"} Oct 30 22:11:16.679521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.679483 4859 scope.go:122] "RemoveContainer" containerID="dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c" Oct 30 22:11:16.681063 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.681038 4859 generic.go:356] "Generic (PLEG): container finished" podID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerID="02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b" exitCode=0 Oct 30 22:11:16.681172 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.681086 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" event={"ID":"3ed1f318-6c24-4904-8a3d-64a71fcab80e","Type":"ContainerDied","Data":"02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b"} Oct 30 22:11:16.681172 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.681100 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" Oct 30 22:11:16.681172 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.681124 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p" event={"ID":"3ed1f318-6c24-4904-8a3d-64a71fcab80e","Type":"ContainerDied","Data":"34d79e68b62a3064c04c543c66443a1d020d0ecd519b564196926ae3d17b2b56"} Oct 30 22:11:16.698610 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.697741 4859 scope.go:122] "RemoveContainer" containerID="dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c" Oct 30 22:11:16.698610 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:11:16.698206 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c\": container with ID starting with dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c not found: ID does not exist" containerID="dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c" Oct 30 22:11:16.698610 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.698239 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c"} err="failed to get container status \"dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c\": rpc error: code = NotFound desc = could not find container \"dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c\": container with ID starting with dbf5341043fc8fbb04ea09a658d350317b96f3c37d9a67496158d55f9308953c not found: ID does not exist" Oct 30 22:11:16.698610 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.698291 4859 scope.go:122] "RemoveContainer" containerID="02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b" Oct 30 22:11:16.715658 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.715630 4859 scope.go:122] "RemoveContainer" containerID="02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b" Oct 30 22:11:16.716076 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:11:16.716047 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b\": container with ID starting with 02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b not found: ID does not exist" containerID="02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b" Oct 30 22:11:16.716176 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.716084 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b"} err="failed to get container status \"02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b\": rpc error: code = NotFound desc = could not find container \"02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b\": container with ID starting with 02df60572374f3162e7e2365ac34d533fb0a3d1c8e813e8a8a32d3fb5ebd294b not found: ID does not exist" Oct 30 22:11:16.846903 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.846821 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd"] Oct 30 22:11:16.855046 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.855017 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z9fkd"] Oct 30 22:11:16.976024 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.975992 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p"] Oct 30 22:11:16.978598 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:16.978577 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fcg9p"] Oct 30 22:11:18.741850 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:18.741813 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" path="/var/lib/kubelet/pods/0dec6bd6-145b-4257-8077-4588a7e68960/volumes" Oct 30 22:11:18.742318 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:18.742302 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" path="/var/lib/kubelet/pods/3ed1f318-6c24-4904-8a3d-64a71fcab80e/volumes" Oct 30 22:11:26.396454 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.396409 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt"] Oct 30 22:11:26.397052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.396760 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" podUID="a2c33609-8347-4c22-bdb9-3201b4785ccc" containerName="maas-api" containerID="cri-o://50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4" gracePeriod=30 Oct 30 22:11:26.715636 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.715606 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:11:26.756791 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.756758 4859 generic.go:356] "Generic (PLEG): container finished" podID="a2c33609-8347-4c22-bdb9-3201b4785ccc" containerID="50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4" exitCode=0 Oct 30 22:11:26.756977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.756803 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" event={"ID":"a2c33609-8347-4c22-bdb9-3201b4785ccc","Type":"ContainerDied","Data":"50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4"} Oct 30 22:11:26.756977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.756807 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" Oct 30 22:11:26.756977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.756841 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt" event={"ID":"a2c33609-8347-4c22-bdb9-3201b4785ccc","Type":"ContainerDied","Data":"453eea80dc2fe292595591f4b85fe529d8f461d533df1ba384f14580f63fd650"} Oct 30 22:11:26.756977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.756865 4859 scope.go:122] "RemoveContainer" containerID="50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4" Oct 30 22:11:26.766691 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.766671 4859 scope.go:122] "RemoveContainer" containerID="50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4" Oct 30 22:11:26.766988 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:11:26.766966 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4\": container with ID starting with 50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4 not found: ID does not exist" containerID="50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4" Oct 30 22:11:26.767070 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.767000 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4"} err="failed to get container status \"50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4\": rpc error: code = NotFound desc = could not find container \"50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4\": container with ID starting with 50f010497e282191ff4f16b620caebe9a179dbb4712add97d180695a7e8b31a4 not found: ID does not exist" Oct 30 22:11:26.797491 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.797452 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c33609-8347-4c22-bdb9-3201b4785ccc-kube-api-access-v9rvv\" (UniqueName: \"kubernetes.io/projected/a2c33609-8347-4c22-bdb9-3201b4785ccc-kube-api-access-v9rvv\") pod \"a2c33609-8347-4c22-bdb9-3201b4785ccc\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " Oct 30 22:11:26.797687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.797548 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-maas-api-tls\") pod \"a2c33609-8347-4c22-bdb9-3201b4785ccc\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " Oct 30 22:11:26.797687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.797644 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c33609-8347-4c22-bdb9-3201b4785ccc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c33609-8347-4c22-bdb9-3201b4785ccc-openshift-service-ca\") pod \"a2c33609-8347-4c22-bdb9-3201b4785ccc\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " Oct 30 22:11:26.797815 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.797730 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-metrics-certs\") pod \"a2c33609-8347-4c22-bdb9-3201b4785ccc\" (UID: \"a2c33609-8347-4c22-bdb9-3201b4785ccc\") " Oct 30 22:11:26.801428 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.801396 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c33609-8347-4c22-bdb9-3201b4785ccc-kube-api-access-v9rvv" pod "a2c33609-8347-4c22-bdb9-3201b4785ccc" (UID: "a2c33609-8347-4c22-bdb9-3201b4785ccc"). InnerVolumeSpecName "kube-api-access-v9rvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:26.811507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.810003 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-maas-api-tls" pod "a2c33609-8347-4c22-bdb9-3201b4785ccc" (UID: "a2c33609-8347-4c22-bdb9-3201b4785ccc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:26.813468 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.812180 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-metrics-certs" pod "a2c33609-8347-4c22-bdb9-3201b4785ccc" (UID: "a2c33609-8347-4c22-bdb9-3201b4785ccc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:26.814429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.814399 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c33609-8347-4c22-bdb9-3201b4785ccc-openshift-service-ca" pod "a2c33609-8347-4c22-bdb9-3201b4785ccc" (UID: "a2c33609-8347-4c22-bdb9-3201b4785ccc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:26.900212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.900175 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:26.900212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.900207 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rvv\" (UniqueName: \"kubernetes.io/projected/a2c33609-8347-4c22-bdb9-3201b4785ccc-kube-api-access-v9rvv\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:26.900212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.900219 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c33609-8347-4c22-bdb9-3201b4785ccc-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:26.900526 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:26.900228 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c33609-8347-4c22-bdb9-3201b4785ccc-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:27.180740 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:27.180707 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt"] Oct 30 22:11:27.185922 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:27.185886 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da0f3d-7987fbd77d-cr5jt"] Oct 30 22:11:28.742409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:28.742374 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c33609-8347-4c22-bdb9-3201b4785ccc" path="/var/lib/kubelet/pods/a2c33609-8347-4c22-bdb9-3201b4785ccc/volumes" Oct 30 22:11:39.878205 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:11:39.878114 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:11:53.319747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:11:53.319707 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:12:09.878367 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:12:09.878327 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:12:23.076515 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:12:23.076473 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:12:23.086776 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:12:23.086741 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:12:23.086969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:12:23.086890 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:12:23.095797 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:12:23.095771 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:12:27.368103 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:12:27.368065 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:12:39.878650 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:12:39.878616 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:13:06.754416 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.754336 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7376-0d58d1"] Oct 30 22:13:06.755170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755141 4859 cpu_manager.go:414] "containerMap: removing container" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" Oct 30 22:13:06.755271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755170 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" Oct 30 22:13:06.755271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755196 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerName="istio-proxy" Oct 30 22:13:06.755271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755204 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerName="istio-proxy" Oct 30 22:13:06.755271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755228 4859 cpu_manager.go:414] "containerMap: removing container" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" Oct 30 22:13:06.755271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755235 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" Oct 30 22:13:06.755271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755257 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" Oct 30 22:13:06.755271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755264 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755281 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755288 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755299 4859 cpu_manager.go:414] "containerMap: removing container" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755307 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755320 4859 cpu_manager.go:414] "containerMap: removing container" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755327 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755337 4859 cpu_manager.go:414] "containerMap: removing container" podUID="a2c33609-8347-4c22-bdb9-3201b4785ccc" containerName="maas-api" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755344 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c33609-8347-4c22-bdb9-3201b4785ccc" containerName="maas-api" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755358 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755365 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755380 4859 cpu_manager.go:414] "containerMap: removing container" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755386 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755403 4859 cpu_manager.go:414] "containerMap: removing container" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755411 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755423 4859 cpu_manager.go:414] "containerMap: removing container" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755430 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755439 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755446 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755584 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="784a3a6f-5419-48d6-a7da-3eb93c205b9b" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755599 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c33609-8347-4c22-bdb9-3201b4785ccc" containerName="maas-api" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755610 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755622 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="92deb8c0-dc69-4cba-af2a-a77ed60be8cc" containerName="istio-proxy" Oct 30 22:13:06.755620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755631 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dec6bd6-145b-4257-8077-4588a7e68960" containerName="istio-proxy" Oct 30 22:13:06.756704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755640 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8edd5bc5-174f-4554-9d7d-a5f0d9e1170e" containerName="istio-proxy" Oct 30 22:13:06.756704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755648 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9293cc9-9ba6-4328-a72d-df484d2388e1" containerName="istio-proxy" Oct 30 22:13:06.756704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755659 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="748bacd4-dd89-4c59-8be2-1a8bae565bb7" containerName="istio-proxy" Oct 30 22:13:06.756704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755671 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed1f318-6c24-4904-8a3d-64a71fcab80e" containerName="istio-proxy" Oct 30 22:13:06.756704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755682 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="0630b556-6459-480b-abf9-a3f3d0ecc6c1" containerName="istio-proxy" Oct 30 22:13:06.756704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755693 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3479d7ff-6792-481a-ad14-0672fb6ff04e" containerName="istio-proxy" Oct 30 22:13:06.756704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.755703 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b698de93-8da9-45f6-954f-c8d7c64118d3" containerName="istio-proxy" Oct 30 22:13:06.759650 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.759628 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-0d58d1" Oct 30 22:13:06.811822 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.811792 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zbdpx\"" Oct 30 22:13:06.816181 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.816154 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7376-0d58d1"] Oct 30 22:13:06.899418 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:06.899206 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wch2t\" (UniqueName: \"kubernetes.io/projected/b00f9fdf-7d9c-46ec-a625-5bf368d83148-kube-api-access-wch2t\") pod \"test-curl-7376-0d58d1\" (UID: \"b00f9fdf-7d9c-46ec-a625-5bf368d83148\") " pod="opendatahub/test-curl-7376-0d58d1" Oct 30 22:13:07.005273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:07.005176 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wch2t\" (UniqueName: \"kubernetes.io/projected/b00f9fdf-7d9c-46ec-a625-5bf368d83148-kube-api-access-wch2t\") pod \"test-curl-7376-0d58d1\" (UID: \"b00f9fdf-7d9c-46ec-a625-5bf368d83148\") " pod="opendatahub/test-curl-7376-0d58d1" Oct 30 22:13:07.062618 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:07.062586 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wch2t\" (UniqueName: \"kubernetes.io/projected/b00f9fdf-7d9c-46ec-a625-5bf368d83148-kube-api-access-wch2t\") pod \"test-curl-7376-0d58d1\" (UID: \"b00f9fdf-7d9c-46ec-a625-5bf368d83148\") " pod="opendatahub/test-curl-7376-0d58d1" Oct 30 22:13:07.072858 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:07.072829 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-0d58d1" Oct 30 22:13:07.474984 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:07.468666 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7376-0d58d1"] Oct 30 22:13:07.502683 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:07.502649 4859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:13:07.525348 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:07.525306 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k"] Oct 30 22:13:07.525636 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:07.525584 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" containerID="cri-o://28793307505ac3efdf69d16e1b95351fad8d4109f142ab77e4351bf511fb39f1" gracePeriod=30 Oct 30 22:13:07.553130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:07.553100 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7376-0d58d1" event={"ID":"b00f9fdf-7d9c-46ec-a625-5bf368d83148","Type":"ContainerStarted","Data":"87d54d8e93c7d32b0067ea1a263949d1a591b3e26888809dc15db20da7a539b8"} Oct 30 22:13:09.653325 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.653292 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c"] Oct 30 22:13:09.654020 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.654004 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f5d232-4abf-4adc-9a62-2aa9be7b9fb7" containerName="istio-proxy" Oct 30 22:13:09.657416 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.657400 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.704612 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.704580 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj"] Oct 30 22:13:09.710739 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.710710 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.745252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.745226 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.745393 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.745264 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.745393 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.745313 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.745393 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.745360 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.745537 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.745394 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.745537 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.745414 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.745537 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.745447 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.745537 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.745470 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.745537 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.745492 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wth79\" (UniqueName: \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-kube-api-access-wth79\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.761070 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.761041 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c"] Oct 30 22:13:09.847482 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847445 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/706b776e-3056-4f6d-9595-08faf30f0a5f-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.847810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847501 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.847810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847539 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48ljf\" (UniqueName: \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-kube-api-access-48ljf\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.847810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847660 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.847810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847715 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.847810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847791 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.848162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847823 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.848162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847855 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.848162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847898 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.848162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847926 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.848162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.847992 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.848162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.848018 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wth79\" (UniqueName: \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-kube-api-access-wth79\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.848162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.848067 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.848162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.848091 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.848162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.848147 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.848609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.848167 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.848609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.848471 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.848609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.848499 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.848609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.848505 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.848609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.848593 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.850254 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.848860 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.850254 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.849545 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.850254 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.849852 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.852725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.852695 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.853748 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.853727 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.867246 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.867222 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj"] Oct 30 22:13:09.900276 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:13:09.899963 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:13:09.907925 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.907859 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.909032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.909003 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wth79\" (UniqueName: \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-kube-api-access-wth79\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:09.949815 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.949781 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/706b776e-3056-4f6d-9595-08faf30f0a5f-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.949834 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48ljf\" (UniqueName: \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-kube-api-access-48ljf\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.949896 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.949976 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.950015 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.950067 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.950089 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.950135 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.950234 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.950338 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.950525 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.950647 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/706b776e-3056-4f6d-9595-08faf30f0a5f-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950862 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.950838 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.950973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.950900 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.953063 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.953018 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.953428 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.953407 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:09.975019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:09.974991 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:10.007841 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.007811 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:10.008713 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.008691 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48ljf\" (UniqueName: \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-kube-api-access-48ljf\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:10.027305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.027013 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:10.421081 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.421031 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c"] Oct 30 22:13:10.438235 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:13:10.433872 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc40986_cce4_406d_bb17_e0b6eeadcf01.slice/crio-ed7fb44b0f74630edb662cb06523f2c06c0a8b984f09c107582047d8503ae5d4 WatchSource:0}: Error finding container ed7fb44b0f74630edb662cb06523f2c06c0a8b984f09c107582047d8503ae5d4: Status 404 returned error can't find the container with id ed7fb44b0f74630edb662cb06523f2c06c0a8b984f09c107582047d8503ae5d4 Oct 30 22:13:10.438235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.436918 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:10.438235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.437039 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:10.438235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.437080 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:10.474338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.474068 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj"] Oct 30 22:13:10.477074 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:13:10.477044 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706b776e_3056_4f6d_9595_08faf30f0a5f.slice/crio-53c77ee1137e8b659f31ab8a8e01718e3a471287b792458e8026e2581bc84218 WatchSource:0}: Error finding container 53c77ee1137e8b659f31ab8a8e01718e3a471287b792458e8026e2581bc84218: Status 404 returned error can't find the container with id 53c77ee1137e8b659f31ab8a8e01718e3a471287b792458e8026e2581bc84218 Oct 30 22:13:10.478939 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.478907 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:10.479037 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.479021 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:10.479103 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.479066 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:10.604866 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.603811 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" event={"ID":"3bc40986-cce4-406d-bb17-e0b6eeadcf01","Type":"ContainerStarted","Data":"0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6"} Oct 30 22:13:10.604866 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.603876 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" event={"ID":"3bc40986-cce4-406d-bb17-e0b6eeadcf01","Type":"ContainerStarted","Data":"ed7fb44b0f74630edb662cb06523f2c06c0a8b984f09c107582047d8503ae5d4"} Oct 30 22:13:10.607508 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.607477 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" event={"ID":"706b776e-3056-4f6d-9595-08faf30f0a5f","Type":"ContainerStarted","Data":"ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea"} Oct 30 22:13:10.607635 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.607529 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" event={"ID":"706b776e-3056-4f6d-9595-08faf30f0a5f","Type":"ContainerStarted","Data":"53c77ee1137e8b659f31ab8a8e01718e3a471287b792458e8026e2581bc84218"} Oct 30 22:13:10.728664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.728560 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" podStartSLOduration=1.728543785 podStartE2EDuration="1.728543785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:10.724199807 +0000 UTC m=+1848.096826577" watchObservedRunningTime="2026-10-30 22:13:10.728543785 +0000 UTC m=+1848.101170535" Oct 30 22:13:10.976250 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:10.976217 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:11.028241 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:11.028207 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:11.030074 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:11.030041 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:13:11.030200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:11.030108 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 22:13:11.644010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:11.643937 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7376-0d58d1" event={"ID":"b00f9fdf-7d9c-46ec-a625-5bf368d83148","Type":"ContainerStarted","Data":"18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84"} Oct 30 22:13:11.760285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:11.760235 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7376-0d58d1" podStartSLOduration=2.251952637 podStartE2EDuration="5.760223031s" totalImagesPullingTime="3.508270394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:11.758621453 +0000 UTC m=+1849.131248197" watchObservedRunningTime="2026-10-30 22:13:11.760223031 +0000 UTC m=+1849.132849781" Oct 30 22:13:11.791968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:11.782386 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" podStartSLOduration=2.7823620399999998 podStartE2EDuration="2.78236204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:10.846244404 +0000 UTC m=+1848.218871152" watchObservedRunningTime="2026-10-30 22:13:11.78236204 +0000 UTC m=+1849.154988789" Oct 30 22:13:11.979619 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:11.979528 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:13:11.979878 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:11.979848 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:13:12.028667 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:12.028629 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:13:12.028929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:12.028899 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 22:13:12.652457 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:12.652427 4859 generic.go:356] "Generic (PLEG): container finished" podID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerID="28793307505ac3efdf69d16e1b95351fad8d4109f142ab77e4351bf511fb39f1" exitCode=0 Oct 30 22:13:12.652651 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:12.652456 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" event={"ID":"08e4a7cd-72c3-447c-94ac-604279c051aa","Type":"ContainerDied","Data":"28793307505ac3efdf69d16e1b95351fad8d4109f142ab77e4351bf511fb39f1"} Oct 30 22:13:12.983239 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:12.983199 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:13:12.983726 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:12.983277 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:13:13.012970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.012331 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:13:13.027563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.027532 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:13:13.027747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.027585 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 22:13:13.048416 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.048386 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e4a7cd-72c3-447c-94ac-604279c051aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e4a7cd-72c3-447c-94ac-604279c051aa-istiod-ca-cert\") pod \"08e4a7cd-72c3-447c-94ac-604279c051aa\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " Oct 30 22:13:13.048568 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.048471 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-kube-api-access-5k7pg\" (UniqueName: \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-kube-api-access-5k7pg\") pod \"08e4a7cd-72c3-447c-94ac-604279c051aa\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " Oct 30 22:13:13.048568 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.048499 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-certs\") pod \"08e4a7cd-72c3-447c-94ac-604279c051aa\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " Oct 30 22:13:13.048568 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.048535 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-podinfo\") pod \"08e4a7cd-72c3-447c-94ac-604279c051aa\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " Oct 30 22:13:13.048752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.048571 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-socket\") pod \"08e4a7cd-72c3-447c-94ac-604279c051aa\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " Oct 30 22:13:13.048752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.048660 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-token\" (UniqueName: \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-token\") pod \"08e4a7cd-72c3-447c-94ac-604279c051aa\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " Oct 30 22:13:13.048752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.048681 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-credential-socket\") pod \"08e4a7cd-72c3-447c-94ac-604279c051aa\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " Oct 30 22:13:13.048752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.048726 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-envoy\") pod \"08e4a7cd-72c3-447c-94ac-604279c051aa\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " Oct 30 22:13:13.048752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.048746 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-data\") pod \"08e4a7cd-72c3-447c-94ac-604279c051aa\" (UID: \"08e4a7cd-72c3-447c-94ac-604279c051aa\") " Oct 30 22:13:13.049048 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.048989 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-socket" pod "08e4a7cd-72c3-447c-94ac-604279c051aa" (UID: "08e4a7cd-72c3-447c-94ac-604279c051aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:13.049973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.049195 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-certs" pod "08e4a7cd-72c3-447c-94ac-604279c051aa" (UID: "08e4a7cd-72c3-447c-94ac-604279c051aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:13.049973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.049211 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-data" pod "08e4a7cd-72c3-447c-94ac-604279c051aa" (UID: "08e4a7cd-72c3-447c-94ac-604279c051aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:13.049973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.049369 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:13.049973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.049404 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:13.049973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.049419 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:13.049973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.049496 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-credential-socket" pod "08e4a7cd-72c3-447c-94ac-604279c051aa" (UID: "08e4a7cd-72c3-447c-94ac-604279c051aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:13.049973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.049781 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e4a7cd-72c3-447c-94ac-604279c051aa-istiod-ca-cert" pod "08e4a7cd-72c3-447c-94ac-604279c051aa" (UID: "08e4a7cd-72c3-447c-94ac-604279c051aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:13.053019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.051060 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-kube-api-access-5k7pg" pod "08e4a7cd-72c3-447c-94ac-604279c051aa" (UID: "08e4a7cd-72c3-447c-94ac-604279c051aa"). InnerVolumeSpecName "kube-api-access-5k7pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:13.053019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.051626 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-token" pod "08e4a7cd-72c3-447c-94ac-604279c051aa" (UID: "08e4a7cd-72c3-447c-94ac-604279c051aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:13.053019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.052186 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-envoy" pod "08e4a7cd-72c3-447c-94ac-604279c051aa" (UID: "08e4a7cd-72c3-447c-94ac-604279c051aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:13.053305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.053278 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-podinfo" pod "08e4a7cd-72c3-447c-94ac-604279c051aa" (UID: "08e4a7cd-72c3-447c-94ac-604279c051aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:13.150252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.150168 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:13.150252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.150195 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e4a7cd-72c3-447c-94ac-604279c051aa-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:13.150252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.150204 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k7pg\" (UniqueName: \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-kube-api-access-5k7pg\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:13.150252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.150214 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:13.150252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.150223 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e4a7cd-72c3-447c-94ac-604279c051aa-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:13.150252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.150231 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e4a7cd-72c3-447c-94ac-604279c051aa-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:13.440870 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.440831 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7376-0d58d1"] Oct 30 22:13:13.444726 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.444693 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7376-0d58d1"] Oct 30 22:13:13.722012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.721069 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" Oct 30 22:13:13.722012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.721202 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" event={"ID":"08e4a7cd-72c3-447c-94ac-604279c051aa","Type":"ContainerDied","Data":"860b2b6d4d47ef6eaff2ae7b4dc84a8730b8dfc67cd5ffb3a7861f3ef9479c05"} Oct 30 22:13:13.722012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.721258 4859 scope.go:122] "RemoveContainer" containerID="28793307505ac3efdf69d16e1b95351fad8d4109f142ab77e4351bf511fb39f1" Oct 30 22:13:13.724963 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.724252 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7376-0d58d1" podUID="b00f9fdf-7d9c-46ec-a625-5bf368d83148" containerName="test-curl-7376-0d58d1" containerID="cri-o://18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84" gracePeriod=2 Oct 30 22:13:13.842327 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.842295 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:13:13.842500 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.842345 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:13.911963 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.911759 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k"] Oct 30 22:13:13.925990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.925147 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2zw5k"] Oct 30 22:13:13.952341 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:13:13.952306 4859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7376-0d58d1\" is forbidden: User \"system:node:ip-10-0-141-104.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-104.us-west-1.compute.internal' and this object" podUID="b00f9fdf-7d9c-46ec-a625-5bf368d83148" pod="opendatahub/test-curl-7376-0d58d1" Oct 30 22:13:13.976390 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.976300 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:13:13.976390 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:13.976376 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:13:14.028413 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:14.028371 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:13:14.028909 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:14.028434 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 22:13:14.741830 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:14.741801 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" path="/var/lib/kubelet/pods/08e4a7cd-72c3-447c-94ac-604279c051aa/volumes" Oct 30 22:13:14.986103 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:14.986061 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:13:14.986383 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:14.986351 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:13:15.028633 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:15.028549 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:13:15.028633 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:15.028604 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 22:13:15.856047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:15.856010 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57"] Oct 30 22:13:15.856556 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:15.856509 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" containerID="cri-o://b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5" gracePeriod=30 Oct 30 22:13:15.982733 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:15.982645 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:13:15.983002 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:15.982973 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:13:16.034372 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.034332 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:13:16.035317 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.035280 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 22:13:16.354195 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.354170 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-0d58d1" Oct 30 22:13:16.448129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.448089 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00f9fdf-7d9c-46ec-a625-5bf368d83148-kube-api-access-wch2t\" (UniqueName: \"kubernetes.io/projected/b00f9fdf-7d9c-46ec-a625-5bf368d83148-kube-api-access-wch2t\") pod \"b00f9fdf-7d9c-46ec-a625-5bf368d83148\" (UID: \"b00f9fdf-7d9c-46ec-a625-5bf368d83148\") " Oct 30 22:13:16.450342 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.450315 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00f9fdf-7d9c-46ec-a625-5bf368d83148-kube-api-access-wch2t" pod "b00f9fdf-7d9c-46ec-a625-5bf368d83148" (UID: "b00f9fdf-7d9c-46ec-a625-5bf368d83148"). InnerVolumeSpecName "kube-api-access-wch2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:16.549518 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.549484 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wch2t\" (UniqueName: \"kubernetes.io/projected/b00f9fdf-7d9c-46ec-a625-5bf368d83148-kube-api-access-wch2t\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:16.746614 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.746562 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00f9fdf-7d9c-46ec-a625-5bf368d83148" path="/var/lib/kubelet/pods/b00f9fdf-7d9c-46ec-a625-5bf368d83148/volumes" Oct 30 22:13:16.754177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.754144 4859 generic.go:356] "Generic (PLEG): container finished" podID="b00f9fdf-7d9c-46ec-a625-5bf368d83148" containerID="18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84" exitCode=137 Oct 30 22:13:16.754347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.754199 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-0d58d1" Oct 30 22:13:16.754429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.754387 4859 scope.go:122] "RemoveContainer" containerID="18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84" Oct 30 22:13:16.770195 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.770171 4859 scope.go:122] "RemoveContainer" containerID="18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84" Oct 30 22:13:16.770563 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:13:16.770534 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84\": container with ID starting with 18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84 not found: ID does not exist" containerID="18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84" Oct 30 22:13:16.770665 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.770572 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84"} err="failed to get container status \"18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84\": rpc error: code = NotFound desc = could not find container \"18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84\": container with ID starting with 18774ee9c757c368dae0233805e107a7e2518b3874221a786a4edc2d63723f84 not found: ID does not exist" Oct 30 22:13:16.975659 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.975574 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:13:16.975659 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:16.975637 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:13:17.028432 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:17.028397 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:13:17.028627 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:17.028453 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 22:13:17.981011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:17.980980 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:17.981410 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:17.981355 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:17.981989 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:17.981938 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:13:18.032457 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:18.032423 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:18.032751 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:18.032729 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:18.033531 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:18.033513 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:13:20.130881 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:20.130847 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c7s57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:13:20.131286 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:20.130901 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:13:21.547875 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.547851 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 22:13:21.633200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633166 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-kube-api-access-pbcz7\" (UniqueName: \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-kube-api-access-pbcz7\") pod \"b6cce801-630f-4f59-8054-5bee59bde387\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " Oct 30 22:13:21.633397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633231 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-certs\") pod \"b6cce801-630f-4f59-8054-5bee59bde387\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " Oct 30 22:13:21.633397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633272 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-data\") pod \"b6cce801-630f-4f59-8054-5bee59bde387\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " Oct 30 22:13:21.633397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633323 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-socket\") pod \"b6cce801-630f-4f59-8054-5bee59bde387\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " Oct 30 22:13:21.633397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633345 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-istio-token\" (UniqueName: \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-istio-token\") pod \"b6cce801-630f-4f59-8054-5bee59bde387\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " Oct 30 22:13:21.633397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633387 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-envoy\") pod \"b6cce801-630f-4f59-8054-5bee59bde387\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " Oct 30 22:13:21.633664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633432 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6cce801-630f-4f59-8054-5bee59bde387-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cce801-630f-4f59-8054-5bee59bde387-istiod-ca-cert\") pod \"b6cce801-630f-4f59-8054-5bee59bde387\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " Oct 30 22:13:21.633664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633485 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6cce801-630f-4f59-8054-5bee59bde387-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cce801-630f-4f59-8054-5bee59bde387-istio-podinfo\") pod \"b6cce801-630f-4f59-8054-5bee59bde387\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " Oct 30 22:13:21.633664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633517 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-credential-socket\") pod \"b6cce801-630f-4f59-8054-5bee59bde387\" (UID: \"b6cce801-630f-4f59-8054-5bee59bde387\") " Oct 30 22:13:21.633664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633574 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-certs" pod "b6cce801-630f-4f59-8054-5bee59bde387" (UID: "b6cce801-630f-4f59-8054-5bee59bde387"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:21.633812 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633676 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-socket" pod "b6cce801-630f-4f59-8054-5bee59bde387" (UID: "b6cce801-630f-4f59-8054-5bee59bde387"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:21.633812 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.633803 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-data" pod "b6cce801-630f-4f59-8054-5bee59bde387" (UID: "b6cce801-630f-4f59-8054-5bee59bde387"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:21.634029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.634006 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-credential-socket" pod "b6cce801-630f-4f59-8054-5bee59bde387" (UID: "b6cce801-630f-4f59-8054-5bee59bde387"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:21.634029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.634025 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cce801-630f-4f59-8054-5bee59bde387-istiod-ca-cert" pod "b6cce801-630f-4f59-8054-5bee59bde387" (UID: "b6cce801-630f-4f59-8054-5bee59bde387"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:21.634232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.634143 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:21.634232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.634167 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:21.634232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.634182 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:21.634232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.634197 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:21.634232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.634214 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cce801-630f-4f59-8054-5bee59bde387-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:21.635579 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.635556 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-kube-api-access-pbcz7" pod "b6cce801-630f-4f59-8054-5bee59bde387" (UID: "b6cce801-630f-4f59-8054-5bee59bde387"). InnerVolumeSpecName "kube-api-access-pbcz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:21.635881 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.635815 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-istio-token" pod "b6cce801-630f-4f59-8054-5bee59bde387" (UID: "b6cce801-630f-4f59-8054-5bee59bde387"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:21.636130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.636109 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-envoy" pod "b6cce801-630f-4f59-8054-5bee59bde387" (UID: "b6cce801-630f-4f59-8054-5bee59bde387"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:21.636130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.636122 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6cce801-630f-4f59-8054-5bee59bde387-istio-podinfo" pod "b6cce801-630f-4f59-8054-5bee59bde387" (UID: "b6cce801-630f-4f59-8054-5bee59bde387"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:21.735097 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.735070 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:21.735097 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.735094 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cce801-630f-4f59-8054-5bee59bde387-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:21.735296 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.735105 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cce801-630f-4f59-8054-5bee59bde387-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:21.735296 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.735119 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbcz7\" (UniqueName: \"kubernetes.io/projected/b6cce801-630f-4f59-8054-5bee59bde387-kube-api-access-pbcz7\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:21.791436 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.791403 4859 generic.go:356] "Generic (PLEG): container finished" podID="b6cce801-630f-4f59-8054-5bee59bde387" containerID="b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5" exitCode=0 Oct 30 22:13:21.791599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.791450 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" Oct 30 22:13:21.791599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.791489 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" event={"ID":"b6cce801-630f-4f59-8054-5bee59bde387","Type":"ContainerDied","Data":"b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5"} Oct 30 22:13:21.791599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.791544 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57" event={"ID":"b6cce801-630f-4f59-8054-5bee59bde387","Type":"ContainerDied","Data":"17d5ebdfb1bf0cba3da07071d1f4f5e582f4e2d9dc35239aa975a80401c9ede2"} Oct 30 22:13:21.791599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.791568 4859 scope.go:122] "RemoveContainer" containerID="b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5" Oct 30 22:13:21.803077 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.803055 4859 scope.go:122] "RemoveContainer" containerID="b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5" Oct 30 22:13:21.803364 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:13:21.803343 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5\": container with ID starting with b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5 not found: ID does not exist" containerID="b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5" Oct 30 22:13:21.803440 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.803369 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5"} err="failed to get container status \"b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5\": rpc error: code = NotFound desc = could not find container \"b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5\": container with ID starting with b9676608bc12056326f550aa43d0682a9bf29ae191078f2e2c558561f7573ed5 not found: ID does not exist" Oct 30 22:13:21.959290 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.959208 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57"] Oct 30 22:13:21.972908 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:21.972881 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c7s57"] Oct 30 22:13:22.745767 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:22.745731 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6cce801-630f-4f59-8054-5bee59bde387" path="/var/lib/kubelet/pods/b6cce801-630f-4f59-8054-5bee59bde387/volumes" Oct 30 22:13:23.581052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:23.581020 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb"] Oct 30 22:13:23.581272 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:23.581208 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" podUID="b7a6df79-f777-4e98-b749-99a27e252613" containerName="maas-api" containerID="cri-o://e31b2f8633574177e7d674d177c8ea8d8d567db4d47aec09dcb1234778471a87" gracePeriod=30 Oct 30 22:13:23.817836 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:23.817793 4859 generic.go:356] "Generic (PLEG): container finished" podID="b7a6df79-f777-4e98-b749-99a27e252613" containerID="e31b2f8633574177e7d674d177c8ea8d8d567db4d47aec09dcb1234778471a87" exitCode=0 Oct 30 22:13:23.818374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:23.818011 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" event={"ID":"b7a6df79-f777-4e98-b749-99a27e252613","Type":"ContainerDied","Data":"e31b2f8633574177e7d674d177c8ea8d8d567db4d47aec09dcb1234778471a87"} Oct 30 22:13:23.906075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:23.906049 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:13:23.979028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:23.978269 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a6df79-f777-4e98-b749-99a27e252613-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a6df79-f777-4e98-b749-99a27e252613-openshift-service-ca\") pod \"b7a6df79-f777-4e98-b749-99a27e252613\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " Oct 30 22:13:23.979028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:23.978332 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a6df79-f777-4e98-b749-99a27e252613-kube-api-access-sctjs\" (UniqueName: \"kubernetes.io/projected/b7a6df79-f777-4e98-b749-99a27e252613-kube-api-access-sctjs\") pod \"b7a6df79-f777-4e98-b749-99a27e252613\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " Oct 30 22:13:23.979028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:23.978435 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-maas-api-tls\") pod \"b7a6df79-f777-4e98-b749-99a27e252613\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " Oct 30 22:13:23.979028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:23.978510 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-metrics-certs\") pod \"b7a6df79-f777-4e98-b749-99a27e252613\" (UID: \"b7a6df79-f777-4e98-b749-99a27e252613\") " Oct 30 22:13:23.980931 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:23.980636 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a6df79-f777-4e98-b749-99a27e252613-openshift-service-ca" pod "b7a6df79-f777-4e98-b749-99a27e252613" (UID: "b7a6df79-f777-4e98-b749-99a27e252613"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:24.000192 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.000132 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-maas-api-tls" pod "b7a6df79-f777-4e98-b749-99a27e252613" (UID: "b7a6df79-f777-4e98-b749-99a27e252613"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:24.011861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.011804 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a6df79-f777-4e98-b749-99a27e252613-kube-api-access-sctjs" pod "b7a6df79-f777-4e98-b749-99a27e252613" (UID: "b7a6df79-f777-4e98-b749-99a27e252613"). InnerVolumeSpecName "kube-api-access-sctjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:24.013171 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.013123 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-metrics-certs" pod "b7a6df79-f777-4e98-b749-99a27e252613" (UID: "b7a6df79-f777-4e98-b749-99a27e252613"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:24.080479 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.080441 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a6df79-f777-4e98-b749-99a27e252613-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:24.080479 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.080479 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sctjs\" (UniqueName: \"kubernetes.io/projected/b7a6df79-f777-4e98-b749-99a27e252613-kube-api-access-sctjs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:24.080684 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.080493 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:24.080684 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.080505 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a6df79-f777-4e98-b749-99a27e252613-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:24.826177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.826147 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" Oct 30 22:13:24.826602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.826190 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb" event={"ID":"b7a6df79-f777-4e98-b749-99a27e252613","Type":"ContainerDied","Data":"79d77131aaa91895b1728351943b94eb3d05a15c828ce00bf49ce25eff8a4cbe"} Oct 30 22:13:24.826602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.826262 4859 scope.go:122] "RemoveContainer" containerID="e31b2f8633574177e7d674d177c8ea8d8d567db4d47aec09dcb1234778471a87" Oct 30 22:13:24.993690 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:24.993653 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb"] Oct 30 22:13:25.059024 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:25.051934 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a8fa-74b9b74b9-gctsb"] Oct 30 22:13:26.742544 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:26.742515 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a6df79-f777-4e98-b749-99a27e252613" path="/var/lib/kubelet/pods/b7a6df79-f777-4e98-b749-99a27e252613/volumes" Oct 30 22:13:32.266225 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:32.266187 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:13:32.677931 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:32.677849 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv"] Oct 30 22:13:32.678226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:32.678196 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" containerID="cri-o://36f6c9341b4db0748e555d07d3e3a70e7d0ece230474705ecc49504782b9f817" gracePeriod=30 Oct 30 22:13:35.148900 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.148861 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k"] Oct 30 22:13:35.149427 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.149140 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" podUID="5d718ff5-697c-4936-8417-c4fe77e2ca6c" containerName="maas-api" containerID="cri-o://47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7" gracePeriod=30 Oct 30 22:13:35.462591 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.462569 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:13:35.518750 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.518726 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d718ff5-697c-4936-8417-c4fe77e2ca6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d718ff5-697c-4936-8417-c4fe77e2ca6c-openshift-service-ca\") pod \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " Oct 30 22:13:35.518874 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.518782 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-metrics-certs\") pod \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " Oct 30 22:13:35.518988 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.518940 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-maas-api-tls\") pod \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " Oct 30 22:13:35.519037 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.519011 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d718ff5-697c-4936-8417-c4fe77e2ca6c-kube-api-access-gk255\" (UniqueName: \"kubernetes.io/projected/5d718ff5-697c-4936-8417-c4fe77e2ca6c-kube-api-access-gk255\") pod \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\" (UID: \"5d718ff5-697c-4936-8417-c4fe77e2ca6c\") " Oct 30 22:13:35.519089 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.519067 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d718ff5-697c-4936-8417-c4fe77e2ca6c-openshift-service-ca" pod "5d718ff5-697c-4936-8417-c4fe77e2ca6c" (UID: "5d718ff5-697c-4936-8417-c4fe77e2ca6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:35.519494 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.519465 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d718ff5-697c-4936-8417-c4fe77e2ca6c-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:35.520937 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.520909 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-metrics-certs" pod "5d718ff5-697c-4936-8417-c4fe77e2ca6c" (UID: "5d718ff5-697c-4936-8417-c4fe77e2ca6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:35.520937 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.520914 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d718ff5-697c-4936-8417-c4fe77e2ca6c-kube-api-access-gk255" pod "5d718ff5-697c-4936-8417-c4fe77e2ca6c" (UID: "5d718ff5-697c-4936-8417-c4fe77e2ca6c"). InnerVolumeSpecName "kube-api-access-gk255". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:35.521094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.520969 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-maas-api-tls" pod "5d718ff5-697c-4936-8417-c4fe77e2ca6c" (UID: "5d718ff5-697c-4936-8417-c4fe77e2ca6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:35.621224 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.621179 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:35.621224 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.621221 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d718ff5-697c-4936-8417-c4fe77e2ca6c-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:35.621224 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.621232 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk255\" (UniqueName: \"kubernetes.io/projected/5d718ff5-697c-4936-8417-c4fe77e2ca6c-kube-api-access-gk255\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:35.916917 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.916883 4859 generic.go:356] "Generic (PLEG): container finished" podID="5d718ff5-697c-4936-8417-c4fe77e2ca6c" containerID="47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7" exitCode=0 Oct 30 22:13:35.917102 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.917050 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" event={"ID":"5d718ff5-697c-4936-8417-c4fe77e2ca6c","Type":"ContainerDied","Data":"47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7"} Oct 30 22:13:35.917102 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.917090 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" event={"ID":"5d718ff5-697c-4936-8417-c4fe77e2ca6c","Type":"ContainerDied","Data":"b5e25931310d09e0a59ce496755db7cb4cc202f7ca85422cb0666415234d36ba"} Oct 30 22:13:35.917176 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.917116 4859 scope.go:122] "RemoveContainer" containerID="47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7" Oct 30 22:13:35.917257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.917245 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k" Oct 30 22:13:35.929517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.929496 4859 scope.go:122] "RemoveContainer" containerID="47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7" Oct 30 22:13:35.929842 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:13:35.929820 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7\": container with ID starting with 47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7 not found: ID does not exist" containerID="47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7" Oct 30 22:13:35.929906 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:35.929849 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7"} err="failed to get container status \"47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7\": rpc error: code = NotFound desc = could not find container \"47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7\": container with ID starting with 47a3542c6cd761374326ba0f5bcc3013455f98e01af27aa7f880cd939cf7efc7 not found: ID does not exist" Oct 30 22:13:36.081610 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:36.081570 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k"] Oct 30 22:13:36.084789 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:36.084763 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-479363-5dd44fc4bf-ml44k"] Oct 30 22:13:36.741829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:36.741793 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d718ff5-697c-4936-8417-c4fe77e2ca6c" path="/var/lib/kubelet/pods/5d718ff5-697c-4936-8417-c4fe77e2ca6c/volumes" Oct 30 22:13:37.934999 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:37.934927 4859 generic.go:356] "Generic (PLEG): container finished" podID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerID="36f6c9341b4db0748e555d07d3e3a70e7d0ece230474705ecc49504782b9f817" exitCode=0 Oct 30 22:13:37.935437 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:37.934976 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" event={"ID":"847babb5-a5d6-42ad-bf35-86b6620bc9d0","Type":"ContainerDied","Data":"36f6c9341b4db0748e555d07d3e3a70e7d0ece230474705ecc49504782b9f817"} Oct 30 22:13:37.935437 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:37.935102 4859 scope.go:122] "RemoveContainer" containerID="af64bd0dddd2ea9c050168cda973986d7f3eb775c759c45a23a13daee23dc2fa" Oct 30 22:13:37.974259 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:37.974229 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:13:38.051742 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.051715 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-envoy\") pod \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " Oct 30 22:13:38.051916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.051751 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-data\") pod \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " Oct 30 22:13:38.051916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.051775 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-socket\") pod \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " Oct 30 22:13:38.051916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.051796 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-podinfo\") pod \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " Oct 30 22:13:38.051916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.051849 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-certs\") pod \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " Oct 30 22:13:38.052185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.051987 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-credential-socket\") pod \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " Oct 30 22:13:38.052185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052012 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istiod-ca-cert\") pod \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " Oct 30 22:13:38.052185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052048 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-data" pod "847babb5-a5d6-42ad-bf35-86b6620bc9d0" (UID: "847babb5-a5d6-42ad-bf35-86b6620bc9d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:38.052185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052082 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-kube-api-access-tfc9d\" (UniqueName: \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-kube-api-access-tfc9d\") pod \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " Oct 30 22:13:38.052185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052118 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-token\" (UniqueName: \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-token\") pod \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\" (UID: \"847babb5-a5d6-42ad-bf35-86b6620bc9d0\") " Oct 30 22:13:38.052185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052124 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-socket" pod "847babb5-a5d6-42ad-bf35-86b6620bc9d0" (UID: "847babb5-a5d6-42ad-bf35-86b6620bc9d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:38.052511 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052355 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-certs" pod "847babb5-a5d6-42ad-bf35-86b6620bc9d0" (UID: "847babb5-a5d6-42ad-bf35-86b6620bc9d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:38.052511 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052408 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istiod-ca-cert" pod "847babb5-a5d6-42ad-bf35-86b6620bc9d0" (UID: "847babb5-a5d6-42ad-bf35-86b6620bc9d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:38.052511 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052475 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-credential-socket" pod "847babb5-a5d6-42ad-bf35-86b6620bc9d0" (UID: "847babb5-a5d6-42ad-bf35-86b6620bc9d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:38.052733 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052706 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:38.052733 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052731 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:38.052854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052741 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:38.052854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052753 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:38.052854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.052765 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:38.053925 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.053904 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-envoy" pod "847babb5-a5d6-42ad-bf35-86b6620bc9d0" (UID: "847babb5-a5d6-42ad-bf35-86b6620bc9d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:38.054294 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.054276 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-kube-api-access-tfc9d" pod "847babb5-a5d6-42ad-bf35-86b6620bc9d0" (UID: "847babb5-a5d6-42ad-bf35-86b6620bc9d0"). InnerVolumeSpecName "kube-api-access-tfc9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:38.054505 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.054483 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-token" pod "847babb5-a5d6-42ad-bf35-86b6620bc9d0" (UID: "847babb5-a5d6-42ad-bf35-86b6620bc9d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:38.054570 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.054487 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-podinfo" pod "847babb5-a5d6-42ad-bf35-86b6620bc9d0" (UID: "847babb5-a5d6-42ad-bf35-86b6620bc9d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:38.154376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.154349 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfc9d\" (UniqueName: \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-kube-api-access-tfc9d\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:38.154376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.154374 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:38.154571 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.154385 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:38.154571 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.154394 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847babb5-a5d6-42ad-bf35-86b6620bc9d0-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:38.943345 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.943306 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" event={"ID":"847babb5-a5d6-42ad-bf35-86b6620bc9d0","Type":"ContainerDied","Data":"338300e231d0a90ef9c7d431c10cb3fb230658f7f2ed16b4d2bbea4fe5e3e1f1"} Oct 30 22:13:38.943762 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.943367 4859 scope.go:122] "RemoveContainer" containerID="36f6c9341b4db0748e555d07d3e3a70e7d0ece230474705ecc49504782b9f817" Oct 30 22:13:38.943762 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:38.943329 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv" Oct 30 22:13:39.124169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:39.124133 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv"] Oct 30 22:13:39.174335 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:39.174300 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-02a8fa-openshift-default-79878d864c-vfmcv"] Oct 30 22:13:39.881510 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:13:39.881470 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:13:40.742663 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:40.742630 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" path="/var/lib/kubelet/pods/847babb5-a5d6-42ad-bf35-86b6620bc9d0/volumes" Oct 30 22:13:45.300579 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:45.300539 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7"] Oct 30 22:13:45.301191 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:45.300821 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" containerID="cri-o://2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693" gracePeriod=30 Oct 30 22:13:47.218027 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.217989 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh"] Oct 30 22:13:47.218807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218786 4859 cpu_manager.go:414] "containerMap: removing container" podUID="5d718ff5-697c-4936-8417-c4fe77e2ca6c" containerName="maas-api" Oct 30 22:13:47.218929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218808 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d718ff5-697c-4936-8417-c4fe77e2ca6c" containerName="maas-api" Oct 30 22:13:47.218929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218827 4859 cpu_manager.go:414] "containerMap: removing container" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" Oct 30 22:13:47.218929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218834 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" Oct 30 22:13:47.218929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218848 4859 cpu_manager.go:414] "containerMap: removing container" podUID="b7a6df79-f777-4e98-b749-99a27e252613" containerName="maas-api" Oct 30 22:13:47.218929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218856 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a6df79-f777-4e98-b749-99a27e252613" containerName="maas-api" Oct 30 22:13:47.218929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218880 4859 cpu_manager.go:414] "containerMap: removing container" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" Oct 30 22:13:47.218929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218889 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" Oct 30 22:13:47.218929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218906 4859 cpu_manager.go:414] "containerMap: removing container" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" Oct 30 22:13:47.218929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218912 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218938 4859 cpu_manager.go:414] "containerMap: removing container" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218958 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218974 4859 cpu_manager.go:414] "containerMap: removing container" podUID="b00f9fdf-7d9c-46ec-a625-5bf368d83148" containerName="test-curl-7376-0d58d1" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.218980 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00f9fdf-7d9c-46ec-a625-5bf368d83148" containerName="test-curl-7376-0d58d1" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.219125 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00f9fdf-7d9c-46ec-a625-5bf368d83148" containerName="test-curl-7376-0d58d1" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.219144 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d718ff5-697c-4936-8417-c4fe77e2ca6c" containerName="maas-api" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.219161 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.219171 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a6df79-f777-4e98-b749-99a27e252613" containerName="maas-api" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.219184 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e4a7cd-72c3-447c-94ac-604279c051aa" containerName="istio-proxy" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.219193 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="847babb5-a5d6-42ad-bf35-86b6620bc9d0" containerName="istio-proxy" Oct 30 22:13:47.219406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.219204 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cce801-630f-4f59-8054-5bee59bde387" containerName="istio-proxy" Oct 30 22:13:47.222709 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.222688 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.275375 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.275337 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-69a1d2-openshift-default-dockercfg-mtzwx\"" Oct 30 22:13:47.292937 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.292908 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-certs\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.293133 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.292968 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-token\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.293133 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.293085 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5b2h\" (UniqueName: \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-kube-api-access-s5b2h\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.293237 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.293195 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-socket\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.293275 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.293236 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-podinfo\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.293275 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.293258 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-envoy\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.293361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.293289 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-data\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.293449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.293433 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-credential-socket\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.293503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.293485 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istiod-ca-cert\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.336166 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.336127 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh"] Oct 30 22:13:47.396847 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.396800 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istiod-ca-cert\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.397233 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.396931 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-certs\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.397461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.397429 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-token\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.397573 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.397502 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5b2h\" (UniqueName: \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-kube-api-access-s5b2h\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.397642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.397588 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-socket\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.397703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.397638 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-podinfo\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.397703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.397673 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-envoy\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.397809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.397694 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-certs\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.397809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.397710 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-data\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.397913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.397831 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-credential-socket\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.398054 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.398029 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-data\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.404970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.398752 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-socket\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.404970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.399518 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-credential-socket\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.404970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.400312 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istiod-ca-cert\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.428780 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.428744 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-podinfo\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.433839 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.433810 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-envoy\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.455678 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.455643 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-token\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.455825 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.455789 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5b2h\" (UniqueName: \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-kube-api-access-s5b2h\") pod \"e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.537142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.537055 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:47.972478 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.972454 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh"] Oct 30 22:13:47.973194 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:13:47.973161 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c25e8c3_71db_44fe_9c39_85ced3c3aa79.slice/crio-ace12f6b346f34ced528d45d6899a63f0623962385f1d246a2ec3ed195d5f09e WatchSource:0}: Error finding container ace12f6b346f34ced528d45d6899a63f0623962385f1d246a2ec3ed195d5f09e: Status 404 returned error can't find the container with id ace12f6b346f34ced528d45d6899a63f0623962385f1d246a2ec3ed195d5f09e Oct 30 22:13:47.975709 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.975674 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:47.975794 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.975758 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:47.975794 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:47.975787 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:48.010202 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:48.010167 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" event={"ID":"3c25e8c3-71db-44fe-9c39-85ced3c3aa79","Type":"ContainerStarted","Data":"ace12f6b346f34ced528d45d6899a63f0623962385f1d246a2ec3ed195d5f09e"} Oct 30 22:13:49.018452 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:49.018414 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" event={"ID":"3c25e8c3-71db-44fe-9c39-85ced3c3aa79","Type":"ContainerStarted","Data":"c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a"} Oct 30 22:13:49.146637 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:49.146584 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" podStartSLOduration=2.146571679 podStartE2EDuration="2.146571679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:49.144361659 +0000 UTC m=+1886.516988406" watchObservedRunningTime="2026-10-30 22:13:49.146571679 +0000 UTC m=+1886.519198426" Oct 30 22:13:49.537741 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:49.537709 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:49.544137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:49.544107 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:50.026097 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.026065 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:50.027036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.027013 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:13:50.577408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.577344 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:13:50.741305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.741265 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-token\" (UniqueName: \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-token\") pod \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " Oct 30 22:13:50.741509 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.741324 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-kube-api-access-wvvd2\" (UniqueName: \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-kube-api-access-wvvd2\") pod \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " Oct 30 22:13:50.741509 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.741365 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istiod-ca-cert\") pod \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " Oct 30 22:13:50.741509 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.741415 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-certs\") pod \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " Oct 30 22:13:50.741509 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.741478 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-envoy\") pod \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " Oct 30 22:13:50.741731 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.741569 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-data\") pod \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " Oct 30 22:13:50.741731 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.741611 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-socket\") pod \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " Oct 30 22:13:50.741731 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.741690 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-podinfo\") pod \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " Oct 30 22:13:50.741897 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.741728 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-credential-socket\") pod \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\" (UID: \"bfad3673-95ee-4fd4-b984-240d47f2e5b5\") " Oct 30 22:13:50.742930 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.742714 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istiod-ca-cert" pod "bfad3673-95ee-4fd4-b984-240d47f2e5b5" (UID: "bfad3673-95ee-4fd4-b984-240d47f2e5b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:50.742930 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.742879 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-data" pod "bfad3673-95ee-4fd4-b984-240d47f2e5b5" (UID: "bfad3673-95ee-4fd4-b984-240d47f2e5b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:50.743133 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.742977 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-socket" pod "bfad3673-95ee-4fd4-b984-240d47f2e5b5" (UID: "bfad3673-95ee-4fd4-b984-240d47f2e5b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:50.743133 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.743057 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-certs" pod "bfad3673-95ee-4fd4-b984-240d47f2e5b5" (UID: "bfad3673-95ee-4fd4-b984-240d47f2e5b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:50.743133 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.743115 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-credential-socket" pod "bfad3673-95ee-4fd4-b984-240d47f2e5b5" (UID: "bfad3673-95ee-4fd4-b984-240d47f2e5b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:50.746484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.746126 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-kube-api-access-wvvd2" pod "bfad3673-95ee-4fd4-b984-240d47f2e5b5" (UID: "bfad3673-95ee-4fd4-b984-240d47f2e5b5"). InnerVolumeSpecName "kube-api-access-wvvd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:50.746484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.746236 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-token" pod "bfad3673-95ee-4fd4-b984-240d47f2e5b5" (UID: "bfad3673-95ee-4fd4-b984-240d47f2e5b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:50.746884 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.746848 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-envoy" pod "bfad3673-95ee-4fd4-b984-240d47f2e5b5" (UID: "bfad3673-95ee-4fd4-b984-240d47f2e5b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:50.747075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.747053 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-podinfo" pod "bfad3673-95ee-4fd4-b984-240d47f2e5b5" (UID: "bfad3673-95ee-4fd4-b984-240d47f2e5b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:50.844200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.844013 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.844200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.844047 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvvd2\" (UniqueName: \"kubernetes.io/projected/bfad3673-95ee-4fd4-b984-240d47f2e5b5-kube-api-access-wvvd2\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.844200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.844065 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.844200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.844079 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.844200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.844097 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.844200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.844110 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.844200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.844125 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.844200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.844139 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfad3673-95ee-4fd4-b984-240d47f2e5b5-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.844200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:50.844153 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad3673-95ee-4fd4-b984-240d47f2e5b5-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.034409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.034372 4859 generic.go:356] "Generic (PLEG): container finished" podID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerID="2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693" exitCode=0 Oct 30 22:13:51.034899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.034418 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" Oct 30 22:13:51.034899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.034453 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" event={"ID":"bfad3673-95ee-4fd4-b984-240d47f2e5b5","Type":"ContainerDied","Data":"2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693"} Oct 30 22:13:51.034899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.034511 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7" event={"ID":"bfad3673-95ee-4fd4-b984-240d47f2e5b5","Type":"ContainerDied","Data":"c3d56b043b602c30acbca1c327c68211d36b600d2eec2077a15b546ed3fa6708"} Oct 30 22:13:51.034899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.034537 4859 scope.go:122] "RemoveContainer" containerID="2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693" Oct 30 22:13:51.046955 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.046923 4859 scope.go:122] "RemoveContainer" containerID="53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e" Oct 30 22:13:51.056851 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.056820 4859 scope.go:122] "RemoveContainer" containerID="2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693" Oct 30 22:13:51.057202 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:13:51.057176 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693\": container with ID starting with 2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693 not found: ID does not exist" containerID="2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693" Oct 30 22:13:51.057285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.057216 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693"} err="failed to get container status \"2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693\": rpc error: code = NotFound desc = could not find container \"2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693\": container with ID starting with 2be6b6e7df07bffcb0564f098d109d40840ea301e7314c8eb34318536f38a693 not found: ID does not exist" Oct 30 22:13:51.057285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.057235 4859 scope.go:122] "RemoveContainer" containerID="53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e" Oct 30 22:13:51.057489 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:13:51.057469 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e\": container with ID starting with 53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e not found: ID does not exist" containerID="53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e" Oct 30 22:13:51.057550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.057494 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e"} err="failed to get container status \"53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e\": rpc error: code = NotFound desc = could not find container \"53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e\": container with ID starting with 53b9dd48f138a02bbb3759cff0abce0a1cf1163f36e362b0a562963f7e46cc8e not found: ID does not exist" Oct 30 22:13:51.201706 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.201607 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7"] Oct 30 22:13:51.250909 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:51.250867 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-479363-openshift-default-78d54cdf47-cp5m7"] Oct 30 22:13:52.742209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:13:52.742176 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" path="/var/lib/kubelet/pods/bfad3673-95ee-4fd4-b984-240d47f2e5b5/volumes" Oct 30 22:14:00.536938 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:00.536901 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:14:09.879367 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:09.879323 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:14:22.558138 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.558105 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh"] Oct 30 22:14:22.558674 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.558659 4859 cpu_manager.go:414] "containerMap: removing container" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" Oct 30 22:14:22.558720 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.558676 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" Oct 30 22:14:22.558720 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.558703 4859 cpu_manager.go:414] "containerMap: removing container" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" Oct 30 22:14:22.558720 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.558708 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" Oct 30 22:14:22.558826 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.558815 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" Oct 30 22:14:22.558863 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.558830 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfad3673-95ee-4fd4-b984-240d47f2e5b5" containerName="istio-proxy" Oct 30 22:14:22.562297 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.562275 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.615144 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.615117 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7a54db-serving-cert\"" Oct 30 22:14:22.631128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.631007 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df95d869-36df-46ea-861c-7a01b29cf84f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.631128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.631069 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.638307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.631201 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjbq\" (UniqueName: \"kubernetes.io/projected/df95d869-36df-46ea-861c-7a01b29cf84f-kube-api-access-kpjbq\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.638307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.631283 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-metrics-certs\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.670517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.670482 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh"] Oct 30 22:14:22.732927 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.732891 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df95d869-36df-46ea-861c-7a01b29cf84f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.732927 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.732932 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.733182 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.733008 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjbq\" (UniqueName: \"kubernetes.io/projected/df95d869-36df-46ea-861c-7a01b29cf84f-kube-api-access-kpjbq\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.733182 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.733054 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-metrics-certs\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.733581 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.733556 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df95d869-36df-46ea-861c-7a01b29cf84f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.735548 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.735520 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-metrics-certs\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.783604 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.783579 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7a54db-serving-cert\"" Oct 30 22:14:22.789506 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:22.789483 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjbq\" (UniqueName: \"kubernetes.io/projected/df95d869-36df-46ea-861c-7a01b29cf84f-kube-api-access-kpjbq\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:22.793451 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:22.793431 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-serving-cert: secret "maas-api-e2e-ipp-a-7a54db-serving-cert" not found Oct 30 22:14:22.793573 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:22.793507 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-maas-api-tls podName:df95d869-36df-46ea-861c-7a01b29cf84f nodeName:}" failed. No retries permitted until 2026-10-30 22:14:23.293489136 +0000 UTC m=+1920.666115875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-maas-api-tls") pod "maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" (UID: "df95d869-36df-46ea-861c-7a01b29cf84f") : secret "maas-api-e2e-ipp-a-7a54db-serving-cert" not found Oct 30 22:14:23.345254 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:23.345219 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:23.347610 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:23.347581 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:23.476155 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:23.476122 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:23.857679 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:23.857651 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh"] Oct 30 22:14:23.858418 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:14:23.858391 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf95d869_36df_46ea_861c_7a01b29cf84f.slice/crio-a9d1bace1367b2d5b4b7bfc97e8b98a6929b7d0e2800626f67ba6be1742a5d9e WatchSource:0}: Error finding container a9d1bace1367b2d5b4b7bfc97e8b98a6929b7d0e2800626f67ba6be1742a5d9e: Status 404 returned error can't find the container with id a9d1bace1367b2d5b4b7bfc97e8b98a6929b7d0e2800626f67ba6be1742a5d9e Oct 30 22:14:24.329786 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:24.329695 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" event={"ID":"df95d869-36df-46ea-861c-7a01b29cf84f","Type":"ContainerStarted","Data":"a9d1bace1367b2d5b4b7bfc97e8b98a6929b7d0e2800626f67ba6be1742a5d9e"} Oct 30 22:14:25.340979 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:25.340922 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" event={"ID":"df95d869-36df-46ea-861c-7a01b29cf84f","Type":"ContainerStarted","Data":"0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2"} Oct 30 22:14:25.341467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:25.341082 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:25.459933 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:25.459876 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" podStartSLOduration=2.580283289 podStartE2EDuration="3.459859636s" totalImagesPullingTime="879.576347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:25.457404856 +0000 UTC m=+1922.830031616" watchObservedRunningTime="2026-10-30 22:14:25.459859636 +0000 UTC m=+1922.832486383" Oct 30 22:14:31.353810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:31.353779 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:14:38.978000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:38.967214 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x"] Oct 30 22:14:38.987658 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:38.987629 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.035088 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.035057 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x"] Oct 30 22:14:39.080011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.079981 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-69a1d2-serving-cert\"" Oct 30 22:14:39.157402 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.157369 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bddb921-606a-42e6-a25b-40f2ee858f94-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.157583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.157424 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-metrics-certs\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.157583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.157551 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwbwz\" (UniqueName: \"kubernetes.io/projected/9bddb921-606a-42e6-a25b-40f2ee858f94-kube-api-access-xwbwz\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.157674 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.157642 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-maas-api-tls\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.259623 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.259537 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bddb921-606a-42e6-a25b-40f2ee858f94-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.259796 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.259626 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-metrics-certs\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.259796 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.259665 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwbwz\" (UniqueName: \"kubernetes.io/projected/9bddb921-606a-42e6-a25b-40f2ee858f94-kube-api-access-xwbwz\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.259796 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.259732 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-maas-api-tls\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.259983 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:39.259902 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-serving-cert: secret "maas-api-e2e-ipp-a-69a1d2-serving-cert" not found Oct 30 22:14:39.260039 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:39.260008 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-maas-api-tls podName:9bddb921-606a-42e6-a25b-40f2ee858f94 nodeName:}" failed. No retries permitted until 2026-10-30 22:14:39.759984966 +0000 UTC m=+1937.132611700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-maas-api-tls") pod "maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" (UID: "9bddb921-606a-42e6-a25b-40f2ee858f94") : secret "maas-api-e2e-ipp-a-69a1d2-serving-cert" not found Oct 30 22:14:39.260424 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.260397 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bddb921-606a-42e6-a25b-40f2ee858f94-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.262491 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.262460 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-metrics-certs\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.319751 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.319719 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwbwz\" (UniqueName: \"kubernetes.io/projected/9bddb921-606a-42e6-a25b-40f2ee858f94-kube-api-access-xwbwz\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.767575 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.767525 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-maas-api-tls\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.770027 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.769992 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-maas-api-tls\") pod \"maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:39.883846 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:39.883797 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:14:39.903426 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:39.903394 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:40.280840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:40.280807 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x"] Oct 30 22:14:40.282626 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:14:40.282597 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bddb921_606a_42e6_a25b_40f2ee858f94.slice/crio-07f617a25f02aa1c4c4d50c88e7f4b92e501bfbeeaddcd275b165f41b61d144e WatchSource:0}: Error finding container 07f617a25f02aa1c4c4d50c88e7f4b92e501bfbeeaddcd275b165f41b61d144e: Status 404 returned error can't find the container with id 07f617a25f02aa1c4c4d50c88e7f4b92e501bfbeeaddcd275b165f41b61d144e Oct 30 22:14:40.451114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:40.451073 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" event={"ID":"9bddb921-606a-42e6-a25b-40f2ee858f94","Type":"ContainerStarted","Data":"07f617a25f02aa1c4c4d50c88e7f4b92e501bfbeeaddcd275b165f41b61d144e"} Oct 30 22:14:40.843027 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:40.842990 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:14:41.461485 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:41.461447 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" event={"ID":"9bddb921-606a-42e6-a25b-40f2ee858f94","Type":"ContainerStarted","Data":"ada3e3ad244b113c8547224fa3b56abd2fd8bcfca2f4d1c63e8be2f4f27f8657"} Oct 30 22:14:41.461971 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:41.461534 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:41.581279 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:41.581161 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" podStartSLOduration=2.631828457 podStartE2EDuration="3.581144687s" totalImagesPullingTime="949.31623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:41.577226466 +0000 UTC m=+1938.949853216" watchObservedRunningTime="2026-10-30 22:14:41.581144687 +0000 UTC m=+1938.953771435" Oct 30 22:14:46.567745 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:46.567710 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2"] Oct 30 22:14:46.568188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:46.567909 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" containerID="cri-o://b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc" gracePeriod=30 Oct 30 22:14:47.001521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:47.001488 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h"] Oct 30 22:14:47.002004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:47.001967 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" containerID="cri-o://02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c" gracePeriod=30 Oct 30 22:14:47.006892 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:47.006862 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj"] Oct 30 22:14:47.007100 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:47.007071 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" containerID="cri-o://ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea" gracePeriod=30 Oct 30 22:14:47.013890 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:47.013860 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c"] Oct 30 22:14:47.014189 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:47.014160 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" containerID="cri-o://0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6" gracePeriod=30 Oct 30 22:14:47.474304 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:47.474272 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:14:47.983827 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:47.983794 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:14:47.984221 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:47.983850 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:14:48.035657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:48.035626 4859 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:14:48.035816 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:48.035678 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:14:50.118825 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:50.118786 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6sd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:14:50.119291 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:50.118852 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:14:51.829308 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.829279 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 22:14:51.927205 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927159 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c8242e4-ae99-448e-a660-a01fe5076393-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8242e4-ae99-448e-a660-a01fe5076393-istio-podinfo\") pod \"8c8242e4-ae99-448e-a660-a01fe5076393\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " Oct 30 22:14:51.927361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927265 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-certs\") pod \"8c8242e4-ae99-448e-a660-a01fe5076393\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " Oct 30 22:14:51.927361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927317 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-credential-socket\") pod \"8c8242e4-ae99-448e-a660-a01fe5076393\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " Oct 30 22:14:51.927486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927378 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8242e4-ae99-448e-a660-a01fe5076393-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8242e4-ae99-448e-a660-a01fe5076393-istiod-ca-cert\") pod \"8c8242e4-ae99-448e-a660-a01fe5076393\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " Oct 30 22:14:51.927486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927419 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-envoy\") pod \"8c8242e4-ae99-448e-a660-a01fe5076393\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " Oct 30 22:14:51.927593 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927498 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-data\") pod \"8c8242e4-ae99-448e-a660-a01fe5076393\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " Oct 30 22:14:51.927593 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927530 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-certs" pod "8c8242e4-ae99-448e-a660-a01fe5076393" (UID: "8c8242e4-ae99-448e-a660-a01fe5076393"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:51.927593 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927553 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-socket\") pod \"8c8242e4-ae99-448e-a660-a01fe5076393\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " Oct 30 22:14:51.927742 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927655 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-credential-socket" pod "8c8242e4-ae99-448e-a660-a01fe5076393" (UID: "8c8242e4-ae99-448e-a660-a01fe5076393"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:51.927838 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927797 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-data" pod "8c8242e4-ae99-448e-a660-a01fe5076393" (UID: "8c8242e4-ae99-448e-a660-a01fe5076393"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:51.927886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927835 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-socket" pod "8c8242e4-ae99-448e-a660-a01fe5076393" (UID: "8c8242e4-ae99-448e-a660-a01fe5076393"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:51.927929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927889 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-istio-token\" (UniqueName: \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-istio-token\") pod \"8c8242e4-ae99-448e-a660-a01fe5076393\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " Oct 30 22:14:51.927989 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.927935 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-kube-api-access-8t94f\" (UniqueName: \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-kube-api-access-8t94f\") pod \"8c8242e4-ae99-448e-a660-a01fe5076393\" (UID: \"8c8242e4-ae99-448e-a660-a01fe5076393\") " Oct 30 22:14:51.928139 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.928113 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8242e4-ae99-448e-a660-a01fe5076393-istiod-ca-cert" pod "8c8242e4-ae99-448e-a660-a01fe5076393" (UID: "8c8242e4-ae99-448e-a660-a01fe5076393"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:51.928519 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.928492 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:51.928642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.928523 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:51.928642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.928539 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8242e4-ae99-448e-a660-a01fe5076393-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:51.928642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.928553 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:51.928642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.928567 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:51.929808 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.929783 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c8242e4-ae99-448e-a660-a01fe5076393-istio-podinfo" pod "8c8242e4-ae99-448e-a660-a01fe5076393" (UID: "8c8242e4-ae99-448e-a660-a01fe5076393"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:51.930059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.930028 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-envoy" pod "8c8242e4-ae99-448e-a660-a01fe5076393" (UID: "8c8242e4-ae99-448e-a660-a01fe5076393"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:51.930293 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.930271 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-kube-api-access-8t94f" pod "8c8242e4-ae99-448e-a660-a01fe5076393" (UID: "8c8242e4-ae99-448e-a660-a01fe5076393"). InnerVolumeSpecName "kube-api-access-8t94f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:51.931078 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:51.931053 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-istio-token" pod "8c8242e4-ae99-448e-a660-a01fe5076393" (UID: "8c8242e4-ae99-448e-a660-a01fe5076393"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:52.029801 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.029772 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8242e4-ae99-448e-a660-a01fe5076393-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.029801 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.029801 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.029995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.029813 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t94f\" (UniqueName: \"kubernetes.io/projected/8c8242e4-ae99-448e-a660-a01fe5076393-kube-api-access-8t94f\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.029995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.029823 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8242e4-ae99-448e-a660-a01fe5076393-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.268207 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.268177 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:14:52.328594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.318701 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:14:52.333534 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.333505 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-podinfo\") pod \"706b776e-3056-4f6d-9595-08faf30f0a5f\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " Oct 30 22:14:52.333705 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.333690 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-socket\") pod \"706b776e-3056-4f6d-9595-08faf30f0a5f\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " Oct 30 22:14:52.333774 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.333732 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-token\" (UniqueName: \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-token\") pod \"706b776e-3056-4f6d-9595-08faf30f0a5f\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " Oct 30 22:14:52.333911 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.333842 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-credential-socket\") pod \"706b776e-3056-4f6d-9595-08faf30f0a5f\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " Oct 30 22:14:52.333911 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.333903 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-data\") pod \"706b776e-3056-4f6d-9595-08faf30f0a5f\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " Oct 30 22:14:52.334034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.333993 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-socket" pod "706b776e-3056-4f6d-9595-08faf30f0a5f" (UID: "706b776e-3056-4f6d-9595-08faf30f0a5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.334034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334006 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-kube-api-access-48ljf\" (UniqueName: \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-kube-api-access-48ljf\") pod \"706b776e-3056-4f6d-9595-08faf30f0a5f\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " Oct 30 22:14:52.334100 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334080 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-certs\") pod \"706b776e-3056-4f6d-9595-08faf30f0a5f\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " Oct 30 22:14:52.334155 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334140 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-envoy\") pod \"706b776e-3056-4f6d-9595-08faf30f0a5f\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " Oct 30 22:14:52.334192 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334172 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706b776e-3056-4f6d-9595-08faf30f0a5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/706b776e-3056-4f6d-9595-08faf30f0a5f-istiod-ca-cert\") pod \"706b776e-3056-4f6d-9595-08faf30f0a5f\" (UID: \"706b776e-3056-4f6d-9595-08faf30f0a5f\") " Oct 30 22:14:52.334242 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334203 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-credential-socket" pod "706b776e-3056-4f6d-9595-08faf30f0a5f" (UID: "706b776e-3056-4f6d-9595-08faf30f0a5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.334521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334496 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-data" pod "706b776e-3056-4f6d-9595-08faf30f0a5f" (UID: "706b776e-3056-4f6d-9595-08faf30f0a5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.334643 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334581 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706b776e-3056-4f6d-9595-08faf30f0a5f-istiod-ca-cert" pod "706b776e-3056-4f6d-9595-08faf30f0a5f" (UID: "706b776e-3056-4f6d-9595-08faf30f0a5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:52.334712 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334651 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-certs" pod "706b776e-3056-4f6d-9595-08faf30f0a5f" (UID: "706b776e-3056-4f6d-9595-08faf30f0a5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.334789 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334770 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.334848 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334794 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.334848 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334809 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.334848 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334822 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/706b776e-3056-4f6d-9595-08faf30f0a5f-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.334848 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.334837 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.336100 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.336079 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-token" pod "706b776e-3056-4f6d-9595-08faf30f0a5f" (UID: "706b776e-3056-4f6d-9595-08faf30f0a5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:52.336310 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.336287 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-envoy" pod "706b776e-3056-4f6d-9595-08faf30f0a5f" (UID: "706b776e-3056-4f6d-9595-08faf30f0a5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.337065 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.337042 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-kube-api-access-48ljf" pod "706b776e-3056-4f6d-9595-08faf30f0a5f" (UID: "706b776e-3056-4f6d-9595-08faf30f0a5f"). InnerVolumeSpecName "kube-api-access-48ljf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:52.337129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.337047 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-podinfo" pod "706b776e-3056-4f6d-9595-08faf30f0a5f" (UID: "706b776e-3056-4f6d-9595-08faf30f0a5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:52.343985 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.343964 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:14:52.435934 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.435853 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6373d52e-4048-41f8-a998-1532d9ed4403-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6373d52e-4048-41f8-a998-1532d9ed4403-istio-podinfo\") pod \"6373d52e-4048-41f8-a998-1532d9ed4403\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " Oct 30 22:14:52.435934 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.435899 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-envoy\") pod \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " Oct 30 22:14:52.435934 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.435921 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-certs\") pod \"6373d52e-4048-41f8-a998-1532d9ed4403\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " Oct 30 22:14:52.436232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.435938 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-socket\") pod \"6373d52e-4048-41f8-a998-1532d9ed4403\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " Oct 30 22:14:52.436232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436002 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-istio-token\" (UniqueName: \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-istio-token\") pod \"6373d52e-4048-41f8-a998-1532d9ed4403\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " Oct 30 22:14:52.436232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436025 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6373d52e-4048-41f8-a998-1532d9ed4403-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6373d52e-4048-41f8-a998-1532d9ed4403-istiod-ca-cert\") pod \"6373d52e-4048-41f8-a998-1532d9ed4403\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " Oct 30 22:14:52.436232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436046 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-credential-socket\") pod \"6373d52e-4048-41f8-a998-1532d9ed4403\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " Oct 30 22:14:52.436452 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436275 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-certs" pod "6373d52e-4048-41f8-a998-1532d9ed4403" (UID: "6373d52e-4048-41f8-a998-1532d9ed4403"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.436452 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436311 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-socket" pod "6373d52e-4048-41f8-a998-1532d9ed4403" (UID: "6373d52e-4048-41f8-a998-1532d9ed4403"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.436452 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436370 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-podinfo\") pod \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " Oct 30 22:14:52.436626 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436581 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-credential-socket" pod "6373d52e-4048-41f8-a998-1532d9ed4403" (UID: "6373d52e-4048-41f8-a998-1532d9ed4403"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.436626 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436588 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6373d52e-4048-41f8-a998-1532d9ed4403-istiod-ca-cert" pod "6373d52e-4048-41f8-a998-1532d9ed4403" (UID: "6373d52e-4048-41f8-a998-1532d9ed4403"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:52.436846 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436821 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-credential-socket\") pod \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " Oct 30 22:14:52.437004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436888 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-credential-socket" pod "3bc40986-cce4-406d-bb17-e0b6eeadcf01" (UID: "3bc40986-cce4-406d-bb17-e0b6eeadcf01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.437004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436929 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istiod-ca-cert\") pod \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " Oct 30 22:14:52.437004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.436986 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-data\") pod \"6373d52e-4048-41f8-a998-1532d9ed4403\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " Oct 30 22:14:52.437209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437032 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-data\") pod \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " Oct 30 22:14:52.437209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437083 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-envoy\") pod \"6373d52e-4048-41f8-a998-1532d9ed4403\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " Oct 30 22:14:52.437209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437164 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-token\" (UniqueName: \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-token\") pod \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " Oct 30 22:14:52.437385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437236 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-socket\") pod \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " Oct 30 22:14:52.437385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437271 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-kube-api-access-wth79\" (UniqueName: \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-kube-api-access-wth79\") pod \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " Oct 30 22:14:52.437385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437308 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-kube-api-access-zhgx4\" (UniqueName: \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-kube-api-access-zhgx4\") pod \"6373d52e-4048-41f8-a998-1532d9ed4403\" (UID: \"6373d52e-4048-41f8-a998-1532d9ed4403\") " Oct 30 22:14:52.437385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437364 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-certs\") pod \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\" (UID: \"3bc40986-cce4-406d-bb17-e0b6eeadcf01\") " Oct 30 22:14:52.437600 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437463 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-data" pod "6373d52e-4048-41f8-a998-1532d9ed4403" (UID: "6373d52e-4048-41f8-a998-1532d9ed4403"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.437780 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437758 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-socket" pod "3bc40986-cce4-406d-bb17-e0b6eeadcf01" (UID: "3bc40986-cce4-406d-bb17-e0b6eeadcf01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.437985 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437966 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438071 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437996 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48ljf\" (UniqueName: \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-kube-api-access-48ljf\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438071 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438012 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438071 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438027 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438071 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438041 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438071 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438056 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438071 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438085 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/706b776e-3056-4f6d-9595-08faf30f0a5f-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438101 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6373d52e-4048-41f8-a998-1532d9ed4403-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438115 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438130 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.438417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.437991 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-certs" pod "3bc40986-cce4-406d-bb17-e0b6eeadcf01" (UID: "3bc40986-cce4-406d-bb17-e0b6eeadcf01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.438417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438185 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istiod-ca-cert" pod "3bc40986-cce4-406d-bb17-e0b6eeadcf01" (UID: "3bc40986-cce4-406d-bb17-e0b6eeadcf01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:52.438796 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438417 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-data" pod "3bc40986-cce4-406d-bb17-e0b6eeadcf01" (UID: "3bc40986-cce4-406d-bb17-e0b6eeadcf01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.438796 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438478 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-envoy" pod "3bc40986-cce4-406d-bb17-e0b6eeadcf01" (UID: "3bc40986-cce4-406d-bb17-e0b6eeadcf01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.438970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438861 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6373d52e-4048-41f8-a998-1532d9ed4403-istio-podinfo" pod "6373d52e-4048-41f8-a998-1532d9ed4403" (UID: "6373d52e-4048-41f8-a998-1532d9ed4403"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:52.438970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.438915 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-podinfo" pod "3bc40986-cce4-406d-bb17-e0b6eeadcf01" (UID: "3bc40986-cce4-406d-bb17-e0b6eeadcf01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:52.439094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.439000 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-istio-token" pod "6373d52e-4048-41f8-a998-1532d9ed4403" (UID: "6373d52e-4048-41f8-a998-1532d9ed4403"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:52.440026 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.440007 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-envoy" pod "6373d52e-4048-41f8-a998-1532d9ed4403" (UID: "6373d52e-4048-41f8-a998-1532d9ed4403"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:52.440124 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.440109 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-token" pod "3bc40986-cce4-406d-bb17-e0b6eeadcf01" (UID: "3bc40986-cce4-406d-bb17-e0b6eeadcf01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:52.440438 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.440420 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-kube-api-access-wth79" pod "3bc40986-cce4-406d-bb17-e0b6eeadcf01" (UID: "3bc40986-cce4-406d-bb17-e0b6eeadcf01"). InnerVolumeSpecName "kube-api-access-wth79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:52.440516 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.440500 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-kube-api-access-zhgx4" pod "6373d52e-4048-41f8-a998-1532d9ed4403" (UID: "6373d52e-4048-41f8-a998-1532d9ed4403"). InnerVolumeSpecName "kube-api-access-zhgx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:52.539288 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539249 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.539467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539288 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.539467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539341 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.539467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539356 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.539467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539373 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6373d52e-4048-41f8-a998-1532d9ed4403-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.539467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539388 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.539467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539402 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wth79\" (UniqueName: \"kubernetes.io/projected/3bc40986-cce4-406d-bb17-e0b6eeadcf01-kube-api-access-wth79\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.539467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539418 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhgx4\" (UniqueName: \"kubernetes.io/projected/6373d52e-4048-41f8-a998-1532d9ed4403-kube-api-access-zhgx4\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.539467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539431 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.539467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539445 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6373d52e-4048-41f8-a998-1532d9ed4403-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.539467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.539458 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc40986-cce4-406d-bb17-e0b6eeadcf01-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:52.542718 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.542687 4859 generic.go:356] "Generic (PLEG): container finished" podID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerID="0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6" exitCode=0 Oct 30 22:14:52.542856 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.542748 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" Oct 30 22:14:52.542856 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.542777 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" event={"ID":"3bc40986-cce4-406d-bb17-e0b6eeadcf01","Type":"ContainerDied","Data":"0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6"} Oct 30 22:14:52.543036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.542865 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c" event={"ID":"3bc40986-cce4-406d-bb17-e0b6eeadcf01","Type":"ContainerDied","Data":"ed7fb44b0f74630edb662cb06523f2c06c0a8b984f09c107582047d8503ae5d4"} Oct 30 22:14:52.543036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.542894 4859 scope.go:122] "RemoveContainer" containerID="0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6" Oct 30 22:14:52.544972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.544916 4859 generic.go:356] "Generic (PLEG): container finished" podID="8c8242e4-ae99-448e-a660-a01fe5076393" containerID="b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc" exitCode=0 Oct 30 22:14:52.545214 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.545180 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" event={"ID":"8c8242e4-ae99-448e-a660-a01fe5076393","Type":"ContainerDied","Data":"b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc"} Oct 30 22:14:52.545330 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.545235 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" event={"ID":"8c8242e4-ae99-448e-a660-a01fe5076393","Type":"ContainerDied","Data":"ec88c127df6e3b17aed6ba09cddaf68dff6e502425d6223ea6e5b95a78685c50"} Oct 30 22:14:52.545330 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.545317 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2" Oct 30 22:14:52.547545 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.547520 4859 generic.go:356] "Generic (PLEG): container finished" podID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerID="ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea" exitCode=0 Oct 30 22:14:52.547657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.547573 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" Oct 30 22:14:52.547725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.547660 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" event={"ID":"706b776e-3056-4f6d-9595-08faf30f0a5f","Type":"ContainerDied","Data":"ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea"} Oct 30 22:14:52.547725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.547702 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj" event={"ID":"706b776e-3056-4f6d-9595-08faf30f0a5f","Type":"ContainerDied","Data":"53c77ee1137e8b659f31ab8a8e01718e3a471287b792458e8026e2581bc84218"} Oct 30 22:14:52.550464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.550427 4859 generic.go:356] "Generic (PLEG): container finished" podID="6373d52e-4048-41f8-a998-1532d9ed4403" containerID="02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c" exitCode=0 Oct 30 22:14:52.550582 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.550465 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" event={"ID":"6373d52e-4048-41f8-a998-1532d9ed4403","Type":"ContainerDied","Data":"02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c"} Oct 30 22:14:52.550582 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.550513 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" event={"ID":"6373d52e-4048-41f8-a998-1532d9ed4403","Type":"ContainerDied","Data":"c50fe97698f33645b4142f1be8e2caabb649b59319ddd496c9cfe4862be3b50c"} Oct 30 22:14:52.554958 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.550755 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h" Oct 30 22:14:52.557698 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.557681 4859 scope.go:122] "RemoveContainer" containerID="0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6" Oct 30 22:14:52.558144 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:52.558117 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6\": container with ID starting with 0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6 not found: ID does not exist" containerID="0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6" Oct 30 22:14:52.558253 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.558152 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6"} err="failed to get container status \"0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6\": rpc error: code = NotFound desc = could not find container \"0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6\": container with ID starting with 0f613598e3ec18bc6e26b98453c6b00898d076beae1b80ed33c5b73fa70052b6 not found: ID does not exist" Oct 30 22:14:52.558253 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.558177 4859 scope.go:122] "RemoveContainer" containerID="b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc" Oct 30 22:14:52.572913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.572895 4859 scope.go:122] "RemoveContainer" containerID="b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc" Oct 30 22:14:52.573251 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:52.573233 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc\": container with ID starting with b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc not found: ID does not exist" containerID="b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc" Oct 30 22:14:52.573336 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.573259 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc"} err="failed to get container status \"b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc\": rpc error: code = NotFound desc = could not find container \"b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc\": container with ID starting with b4eff2e4686b1ef6c0548c3cdb98de3efb09fd4ebad3624396c6adbb74f082cc not found: ID does not exist" Oct 30 22:14:52.573336 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.573279 4859 scope.go:122] "RemoveContainer" containerID="ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea" Oct 30 22:14:52.582275 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.582257 4859 scope.go:122] "RemoveContainer" containerID="ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea" Oct 30 22:14:52.582508 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:52.582488 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea\": container with ID starting with ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea not found: ID does not exist" containerID="ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea" Oct 30 22:14:52.582583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.582516 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea"} err="failed to get container status \"ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea\": rpc error: code = NotFound desc = could not find container \"ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea\": container with ID starting with ec49722c16c9a80726dcdd6bf2e2e578f912d66bd1b48143ed460a167fe2c0ea not found: ID does not exist" Oct 30 22:14:52.582583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.582538 4859 scope.go:122] "RemoveContainer" containerID="02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c" Oct 30 22:14:52.591102 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.591084 4859 scope.go:122] "RemoveContainer" containerID="a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269" Oct 30 22:14:52.637332 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.637307 4859 scope.go:122] "RemoveContainer" containerID="02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c" Oct 30 22:14:52.637699 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:52.637674 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c\": container with ID starting with 02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c not found: ID does not exist" containerID="02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c" Oct 30 22:14:52.637802 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.637709 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c"} err="failed to get container status \"02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c\": rpc error: code = NotFound desc = could not find container \"02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c\": container with ID starting with 02eb20a03f02388ecbb41b340660888bcc86f8bfead5c464562f076f1b64de3c not found: ID does not exist" Oct 30 22:14:52.637802 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.637736 4859 scope.go:122] "RemoveContainer" containerID="a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269" Oct 30 22:14:52.638194 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:14:52.638173 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269\": container with ID starting with a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269 not found: ID does not exist" containerID="a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269" Oct 30 22:14:52.638314 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.638210 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269"} err="failed to get container status \"a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269\": rpc error: code = NotFound desc = could not find container \"a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269\": container with ID starting with a422f8bf224a2fa98544a4102c3dfed3a20bfdcfd3c31b082a2ea4627c6be269 not found: ID does not exist" Oct 30 22:14:52.747356 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.747283 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj"] Oct 30 22:14:52.747356 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.747331 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-pqfdj"] Oct 30 22:14:52.852467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.852433 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c"] Oct 30 22:14:52.855717 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.855690 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-88n5c"] Oct 30 22:14:52.963529 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.963497 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2"] Oct 30 22:14:52.968881 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:52.968853 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6sd2"] Oct 30 22:14:53.076825 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:53.076738 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h"] Oct 30 22:14:53.080188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:53.080163 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4cs6h"] Oct 30 22:14:54.742195 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:54.742161 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" path="/var/lib/kubelet/pods/3bc40986-cce4-406d-bb17-e0b6eeadcf01/volumes" Oct 30 22:14:54.742623 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:54.742609 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" path="/var/lib/kubelet/pods/6373d52e-4048-41f8-a998-1532d9ed4403/volumes" Oct 30 22:14:54.743115 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:54.743097 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" path="/var/lib/kubelet/pods/706b776e-3056-4f6d-9595-08faf30f0a5f/volumes" Oct 30 22:14:54.743830 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:14:54.743814 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" path="/var/lib/kubelet/pods/8c8242e4-ae99-448e-a660-a01fe5076393/volumes" Oct 30 22:15:00.222068 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222031 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr"] Oct 30 22:15:00.222636 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222622 4859 cpu_manager.go:414] "containerMap: removing container" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" Oct 30 22:15:00.222680 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222637 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" Oct 30 22:15:00.222680 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222653 4859 cpu_manager.go:414] "containerMap: removing container" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" Oct 30 22:15:00.222680 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222658 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" Oct 30 22:15:00.222680 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222680 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" Oct 30 22:15:00.222812 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222685 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" Oct 30 22:15:00.222812 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222694 4859 cpu_manager.go:414] "containerMap: removing container" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" Oct 30 22:15:00.222812 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222699 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" Oct 30 22:15:00.222812 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222716 4859 cpu_manager.go:414] "containerMap: removing container" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" Oct 30 22:15:00.222812 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222721 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" Oct 30 22:15:00.222812 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222798 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc40986-cce4-406d-bb17-e0b6eeadcf01" containerName="istio-proxy" Oct 30 22:15:00.222812 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222807 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8242e4-ae99-448e-a660-a01fe5076393" containerName="istio-proxy" Oct 30 22:15:00.223045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222816 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="706b776e-3056-4f6d-9595-08faf30f0a5f" containerName="istio-proxy" Oct 30 22:15:00.223045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222824 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" Oct 30 22:15:00.223045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.222832 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6373d52e-4048-41f8-a998-1532d9ed4403" containerName="istio-proxy" Oct 30 22:15:00.228097 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.228078 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" Oct 30 22:15:00.259154 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.259128 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv"] Oct 30 22:15:00.263209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.263185 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" Oct 30 22:15:00.339233 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.339195 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr"] Oct 30 22:15:00.339394 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.339273 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv"] Oct 30 22:15:00.373120 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.373083 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd876fc0-4344-4528-ad3b-8501e277b5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-g52nr\" (UID: \"dd876fc0-4344-4528-ad3b-8501e277b5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" Oct 30 22:15:00.373321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.373162 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqlrm\" (UniqueName: \"kubernetes.io/projected/f959ab35-db85-43c1-882f-d6c708b3a7c7-kube-api-access-cqlrm\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv\" (UID: \"f959ab35-db85-43c1-882f-d6c708b3a7c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" Oct 30 22:15:00.373405 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.373376 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59jnv\" (UniqueName: \"kubernetes.io/projected/dd876fc0-4344-4528-ad3b-8501e277b5df-kube-api-access-59jnv\") pod \"maas-api-key-cleanup-29889975-g52nr\" (UID: \"dd876fc0-4344-4528-ad3b-8501e277b5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" Oct 30 22:15:00.373518 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.373499 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f959ab35-db85-43c1-882f-d6c708b3a7c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv\" (UID: \"f959ab35-db85-43c1-882f-d6c708b3a7c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" Oct 30 22:15:00.379262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.379236 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\"" Oct 30 22:15:00.491021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.490268 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqlrm\" (UniqueName: \"kubernetes.io/projected/f959ab35-db85-43c1-882f-d6c708b3a7c7-kube-api-access-cqlrm\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv\" (UID: \"f959ab35-db85-43c1-882f-d6c708b3a7c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" Oct 30 22:15:00.491021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.490469 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59jnv\" (UniqueName: \"kubernetes.io/projected/dd876fc0-4344-4528-ad3b-8501e277b5df-kube-api-access-59jnv\") pod \"maas-api-key-cleanup-29889975-g52nr\" (UID: \"dd876fc0-4344-4528-ad3b-8501e277b5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" Oct 30 22:15:00.491021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.490557 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f959ab35-db85-43c1-882f-d6c708b3a7c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv\" (UID: \"f959ab35-db85-43c1-882f-d6c708b3a7c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" Oct 30 22:15:00.491021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.490620 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd876fc0-4344-4528-ad3b-8501e277b5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-g52nr\" (UID: \"dd876fc0-4344-4528-ad3b-8501e277b5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" Oct 30 22:15:00.515059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.491474 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd876fc0-4344-4528-ad3b-8501e277b5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-g52nr\" (UID: \"dd876fc0-4344-4528-ad3b-8501e277b5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" Oct 30 22:15:00.515059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.492574 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f959ab35-db85-43c1-882f-d6c708b3a7c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv\" (UID: \"f959ab35-db85-43c1-882f-d6c708b3a7c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" Oct 30 22:15:00.555862 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.555826 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqlrm\" (UniqueName: \"kubernetes.io/projected/f959ab35-db85-43c1-882f-d6c708b3a7c7-kube-api-access-cqlrm\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv\" (UID: \"f959ab35-db85-43c1-882f-d6c708b3a7c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" Oct 30 22:15:00.556038 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.555936 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59jnv\" (UniqueName: \"kubernetes.io/projected/dd876fc0-4344-4528-ad3b-8501e277b5df-kube-api-access-59jnv\") pod \"maas-api-key-cleanup-29889975-g52nr\" (UID: \"dd876fc0-4344-4528-ad3b-8501e277b5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" Oct 30 22:15:00.586239 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.586142 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" Oct 30 22:15:00.841849 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.841818 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" Oct 30 22:15:00.992365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:00.992334 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv"] Oct 30 22:15:00.996931 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:15:00.996900 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf959ab35_db85_43c1_882f_d6c708b3a7c7.slice/crio-f92c3ef3b04b91d9f00403811414ebdd2256c04e0576f369a0d1ee8ec3b16d8f WatchSource:0}: Error finding container f92c3ef3b04b91d9f00403811414ebdd2256c04e0576f369a0d1ee8ec3b16d8f: Status 404 returned error can't find the container with id f92c3ef3b04b91d9f00403811414ebdd2256c04e0576f369a0d1ee8ec3b16d8f Oct 30 22:15:01.286031 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:15:01.283494 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd876fc0_4344_4528_ad3b_8501e277b5df.slice/crio-62fb5e28ee2aa99c11a78af9413f8e999ec17db99e6c0de331bae7ba2d971384 WatchSource:0}: Error finding container 62fb5e28ee2aa99c11a78af9413f8e999ec17db99e6c0de331bae7ba2d971384: Status 404 returned error can't find the container with id 62fb5e28ee2aa99c11a78af9413f8e999ec17db99e6c0de331bae7ba2d971384 Oct 30 22:15:01.288850 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:01.288821 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr"] Oct 30 22:15:01.631029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:01.630989 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" event={"ID":"f959ab35-db85-43c1-882f-d6c708b3a7c7","Type":"ContainerStarted","Data":"8f4efb20bc458614f343e60ecae87e1fca9ab8d5f1c0cf0bd5b415fc1f8cce1c"} Oct 30 22:15:01.631235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:01.631081 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" event={"ID":"f959ab35-db85-43c1-882f-d6c708b3a7c7","Type":"ContainerStarted","Data":"f92c3ef3b04b91d9f00403811414ebdd2256c04e0576f369a0d1ee8ec3b16d8f"} Oct 30 22:15:01.632613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:01.632584 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" event={"ID":"dd876fc0-4344-4528-ad3b-8501e277b5df","Type":"ContainerStarted","Data":"62fb5e28ee2aa99c11a78af9413f8e999ec17db99e6c0de331bae7ba2d971384"} Oct 30 22:15:01.752565 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:01.752476 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" podStartSLOduration=1.7524629680000001 podStartE2EDuration="1.752462968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.748893063 +0000 UTC m=+1959.121519810" watchObservedRunningTime="2026-10-30 22:15:01.752462968 +0000 UTC m=+1959.125089715" Oct 30 22:15:02.641346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:02.641307 4859 generic.go:356] "Generic (PLEG): container finished" podID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerID="14780e24308d5698ce79881740ce00db990e1f90cafa2e7b04da9dbfef53b9cb" exitCode=60 Oct 30 22:15:02.641811 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:02.641385 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" event={"ID":"dd876fc0-4344-4528-ad3b-8501e277b5df","Type":"ContainerDied","Data":"14780e24308d5698ce79881740ce00db990e1f90cafa2e7b04da9dbfef53b9cb"} Oct 30 22:15:02.641811 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:02.641682 4859 scope.go:122] "RemoveContainer" containerID="14780e24308d5698ce79881740ce00db990e1f90cafa2e7b04da9dbfef53b9cb" Oct 30 22:15:02.643442 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:02.643415 4859 generic.go:356] "Generic (PLEG): container finished" podID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerID="8f4efb20bc458614f343e60ecae87e1fca9ab8d5f1c0cf0bd5b415fc1f8cce1c" exitCode=60 Oct 30 22:15:02.643531 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:02.643450 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" event={"ID":"f959ab35-db85-43c1-882f-d6c708b3a7c7","Type":"ContainerDied","Data":"8f4efb20bc458614f343e60ecae87e1fca9ab8d5f1c0cf0bd5b415fc1f8cce1c"} Oct 30 22:15:02.643756 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:02.643740 4859 scope.go:122] "RemoveContainer" containerID="8f4efb20bc458614f343e60ecae87e1fca9ab8d5f1c0cf0bd5b415fc1f8cce1c" Oct 30 22:15:03.652406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:03.652362 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" event={"ID":"f959ab35-db85-43c1-882f-d6c708b3a7c7","Type":"ContainerStarted","Data":"50d4300fc64bcb1231382446a58612760326c6cb6dcf32b9c3257e5a4b1d9004"} Oct 30 22:15:03.654204 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:03.654176 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" event={"ID":"dd876fc0-4344-4528-ad3b-8501e277b5df","Type":"ContainerStarted","Data":"45553b26a39da4a9afca49b9c968c530f63a156c165059d57707055545bb247f"} Oct 30 22:15:03.900656 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:03.900599 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" podStartSLOduration=3.900584029 podStartE2EDuration="3.900584029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.899146743 +0000 UTC m=+1961.271773491" watchObservedRunningTime="2026-10-30 22:15:03.900584029 +0000 UTC m=+1961.273210776" Oct 30 22:15:04.682969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:04.677797 4859 generic.go:356] "Generic (PLEG): container finished" podID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerID="50d4300fc64bcb1231382446a58612760326c6cb6dcf32b9c3257e5a4b1d9004" exitCode=60 Oct 30 22:15:04.682969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:04.677931 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" event={"ID":"f959ab35-db85-43c1-882f-d6c708b3a7c7","Type":"ContainerDied","Data":"50d4300fc64bcb1231382446a58612760326c6cb6dcf32b9c3257e5a4b1d9004"} Oct 30 22:15:04.682969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:04.678014 4859 scope.go:122] "RemoveContainer" containerID="8f4efb20bc458614f343e60ecae87e1fca9ab8d5f1c0cf0bd5b415fc1f8cce1c" Oct 30 22:15:04.682969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:04.678473 4859 scope.go:122] "RemoveContainer" containerID="50d4300fc64bcb1231382446a58612760326c6cb6dcf32b9c3257e5a4b1d9004" Oct 30 22:15:04.682969 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:15:04.678796 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv_odh-ai-gateway-infra(f959ab35-db85-43c1-882f-d6c708b3a7c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" Oct 30 22:15:04.685953 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:04.685904 4859 generic.go:356] "Generic (PLEG): container finished" podID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerID="45553b26a39da4a9afca49b9c968c530f63a156c165059d57707055545bb247f" exitCode=60 Oct 30 22:15:04.686244 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:04.686207 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" event={"ID":"dd876fc0-4344-4528-ad3b-8501e277b5df","Type":"ContainerDied","Data":"45553b26a39da4a9afca49b9c968c530f63a156c165059d57707055545bb247f"} Oct 30 22:15:04.686867 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:04.686849 4859 scope.go:122] "RemoveContainer" containerID="45553b26a39da4a9afca49b9c968c530f63a156c165059d57707055545bb247f" Oct 30 22:15:04.687408 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:15:04.687370 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-g52nr_odh-ai-gateway-infra(dd876fc0-4344-4528-ad3b-8501e277b5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" Oct 30 22:15:04.707050 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:04.707028 4859 scope.go:122] "RemoveContainer" containerID="14780e24308d5698ce79881740ce00db990e1f90cafa2e7b04da9dbfef53b9cb" Oct 30 22:15:05.710516 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:05.710480 4859 scope.go:122] "RemoveContainer" containerID="45553b26a39da4a9afca49b9c968c530f63a156c165059d57707055545bb247f" Oct 30 22:15:05.711043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:05.710565 4859 scope.go:122] "RemoveContainer" containerID="50d4300fc64bcb1231382446a58612760326c6cb6dcf32b9c3257e5a4b1d9004" Oct 30 22:15:05.711043 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:15:05.710973 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv_odh-ai-gateway-infra(f959ab35-db85-43c1-882f-d6c708b3a7c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" Oct 30 22:15:05.711211 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:15:05.710827 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-g52nr_odh-ai-gateway-infra(dd876fc0-4344-4528-ad3b-8501e277b5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" Oct 30 22:15:09.742563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:09.742531 4859 generic.go:356] "Generic (PLEG): container finished" podID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerID="c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a" exitCode=0 Oct 30 22:15:09.743031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:09.742607 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" event={"ID":"3c25e8c3-71db-44fe-9c39-85ced3c3aa79","Type":"ContainerDied","Data":"c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a"} Oct 30 22:15:09.743031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:09.743018 4859 scope.go:122] "RemoveContainer" containerID="c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a" Oct 30 22:15:09.744462 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:09.744431 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:09.744555 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:09.744495 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:09.744555 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:09.744529 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:09.893352 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:15:09.893302 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:15:09.929424 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:09.926618 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5"] Oct 30 22:15:09.932768 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:09.932742 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.004488 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.003353 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5"] Oct 30 22:15:10.020781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.019858 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-data\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.020781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.019968 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-envoy\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.020781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.020033 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-certs\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.020781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.020085 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60d248-a783-420e-a309-16db1b7ee00e-istiod-ca-cert\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.020781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.020140 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60d248-a783-420e-a309-16db1b7ee00e-istio-podinfo\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.020781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.020178 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-istio-token\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.020781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.020225 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-credential-socket\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.020781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.020261 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-socket\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.020781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.020295 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hm9\" (UniqueName: \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-kube-api-access-g4hm9\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.041366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.041329 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2edb49-openshift-default-dockercfg-84296\"" Oct 30 22:15:10.149017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.148980 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-certs\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.149212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.149057 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60d248-a783-420e-a309-16db1b7ee00e-istiod-ca-cert\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.149212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.149120 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60d248-a783-420e-a309-16db1b7ee00e-istio-podinfo\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.149212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.149161 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-istio-token\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.149352 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.149214 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-credential-socket\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.149352 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.149258 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-socket\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.149352 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.149293 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hm9\" (UniqueName: \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-kube-api-access-g4hm9\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.149450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.149373 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-data\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.149484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.149446 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-envoy\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.150906 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.150386 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-certs\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.150906 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.150410 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-credential-socket\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.151418 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.151402 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-socket\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.151774 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.151558 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60d248-a783-420e-a309-16db1b7ee00e-istiod-ca-cert\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.151864 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.151728 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-data\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.154258 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.154230 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-envoy\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.154754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.154732 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60d248-a783-420e-a309-16db1b7ee00e-istio-podinfo\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.207511 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.207477 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-istio-token\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.207721 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.207697 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hm9\" (UniqueName: \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-kube-api-access-g4hm9\") pod \"e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.269389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.269300 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:10.713111 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.713072 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5"] Oct 30 22:15:10.752001 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.751973 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" event={"ID":"3c25e8c3-71db-44fe-9c39-85ced3c3aa79","Type":"ContainerStarted","Data":"321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049"} Oct 30 22:15:10.752605 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:15:10.752581 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d60d248_a783_420e_a309_16db1b7ee00e.slice/crio-2ceefda054f3eaf74ad92a7346f4dbb59f2346a9aef746dc6dd06e8cf2336f98 WatchSource:0}: Error finding container 2ceefda054f3eaf74ad92a7346f4dbb59f2346a9aef746dc6dd06e8cf2336f98: Status 404 returned error can't find the container with id 2ceefda054f3eaf74ad92a7346f4dbb59f2346a9aef746dc6dd06e8cf2336f98 Oct 30 22:15:10.755353 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.755317 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:10.755442 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.755383 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:10.755442 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:10.755409 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:11.537541 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:11.537502 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:15:11.537732 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:11.537681 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:15:11.537781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:11.537733 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:15:11.760120 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:11.760086 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" event={"ID":"9d60d248-a783-420e-a309-16db1b7ee00e","Type":"ContainerStarted","Data":"a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7"} Oct 30 22:15:11.760120 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:11.760132 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" event={"ID":"9d60d248-a783-420e-a309-16db1b7ee00e","Type":"ContainerStarted","Data":"2ceefda054f3eaf74ad92a7346f4dbb59f2346a9aef746dc6dd06e8cf2336f98"} Oct 30 22:15:11.886886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:11.886816 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" podStartSLOduration=2.886796801 podStartE2EDuration="2.886796801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:11.886308026 +0000 UTC m=+1969.258934774" watchObservedRunningTime="2026-10-30 22:15:11.886796801 +0000 UTC m=+1969.259423549" Oct 30 22:15:12.278036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:12.277451 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:12.287274 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:12.287245 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:12.538399 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:12.538314 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:15:12.538399 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:12.538385 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:15:12.789052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:12.788938 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:12.789977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:12.789929 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:15:13.537809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:13.537766 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:15:13.538025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:13.537818 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:15:14.537787 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:14.537754 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:15:14.538222 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:14.537806 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:15:15.537631 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:15.537598 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:15:15.537804 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:15.537650 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:15:16.542596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:16.542566 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:15:16.543074 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:16.543051 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:15:16.543643 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:16.543623 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:15:17.733775 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:17.733737 4859 scope.go:122] "RemoveContainer" containerID="45553b26a39da4a9afca49b9c968c530f63a156c165059d57707055545bb247f" Oct 30 22:15:17.734271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:17.733837 4859 scope.go:122] "RemoveContainer" containerID="50d4300fc64bcb1231382446a58612760326c6cb6dcf32b9c3257e5a4b1d9004" Oct 30 22:15:18.838711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:18.838686 4859 generic.go:356] "Generic (PLEG): container finished" podID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerID="cc199145d6ca302cff3a3ac80f5341e8722809944c4478c21ceb467be55b373a" exitCode=60 Oct 30 22:15:18.839218 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:18.838756 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" event={"ID":"f959ab35-db85-43c1-882f-d6c708b3a7c7","Type":"ContainerDied","Data":"cc199145d6ca302cff3a3ac80f5341e8722809944c4478c21ceb467be55b373a"} Oct 30 22:15:18.839218 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:18.838816 4859 scope.go:122] "RemoveContainer" containerID="50d4300fc64bcb1231382446a58612760326c6cb6dcf32b9c3257e5a4b1d9004" Oct 30 22:15:18.839338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:18.839299 4859 scope.go:122] "RemoveContainer" containerID="cc199145d6ca302cff3a3ac80f5341e8722809944c4478c21ceb467be55b373a" Oct 30 22:15:18.839628 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:15:18.839602 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv_odh-ai-gateway-infra(f959ab35-db85-43c1-882f-d6c708b3a7c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" Oct 30 22:15:18.841119 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:18.841098 4859 generic.go:356] "Generic (PLEG): container finished" podID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerID="cce35cfc7dc8f49c401111db9a67b7bedc662ec53e70ee3b78c3f3df51edb872" exitCode=60 Oct 30 22:15:18.841217 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:18.841166 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" event={"ID":"dd876fc0-4344-4528-ad3b-8501e277b5df","Type":"ContainerDied","Data":"cce35cfc7dc8f49c401111db9a67b7bedc662ec53e70ee3b78c3f3df51edb872"} Oct 30 22:15:18.841534 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:18.841517 4859 scope.go:122] "RemoveContainer" containerID="cce35cfc7dc8f49c401111db9a67b7bedc662ec53e70ee3b78c3f3df51edb872" Oct 30 22:15:18.841796 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:15:18.841780 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-g52nr_odh-ai-gateway-infra(dd876fc0-4344-4528-ad3b-8501e277b5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" Oct 30 22:15:18.852475 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:18.852457 4859 scope.go:122] "RemoveContainer" containerID="45553b26a39da4a9afca49b9c968c530f63a156c165059d57707055545bb247f" Oct 30 22:15:19.965301 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:19.965265 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv"] Oct 30 22:15:20.122802 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.122773 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr"] Oct 30 22:15:20.154789 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.154759 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" Oct 30 22:15:20.216229 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.216151 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f959ab35-db85-43c1-882f-d6c708b3a7c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f959ab35-db85-43c1-882f-d6c708b3a7c7-maas-api-service-ca\") pod \"f959ab35-db85-43c1-882f-d6c708b3a7c7\" (UID: \"f959ab35-db85-43c1-882f-d6c708b3a7c7\") " Oct 30 22:15:20.216229 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.216220 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f959ab35-db85-43c1-882f-d6c708b3a7c7-kube-api-access-cqlrm\" (UniqueName: \"kubernetes.io/projected/f959ab35-db85-43c1-882f-d6c708b3a7c7-kube-api-access-cqlrm\") pod \"f959ab35-db85-43c1-882f-d6c708b3a7c7\" (UID: \"f959ab35-db85-43c1-882f-d6c708b3a7c7\") " Oct 30 22:15:20.216616 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.216584 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f959ab35-db85-43c1-882f-d6c708b3a7c7-maas-api-service-ca" pod "f959ab35-db85-43c1-882f-d6c708b3a7c7" (UID: "f959ab35-db85-43c1-882f-d6c708b3a7c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:20.218684 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.218651 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f959ab35-db85-43c1-882f-d6c708b3a7c7-kube-api-access-cqlrm" pod "f959ab35-db85-43c1-882f-d6c708b3a7c7" (UID: "f959ab35-db85-43c1-882f-d6c708b3a7c7"). InnerVolumeSpecName "kube-api-access-cqlrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:20.290386 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.290347 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" Oct 30 22:15:20.317596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.317562 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd876fc0-4344-4528-ad3b-8501e277b5df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd876fc0-4344-4528-ad3b-8501e277b5df-maas-api-service-ca\") pod \"dd876fc0-4344-4528-ad3b-8501e277b5df\" (UID: \"dd876fc0-4344-4528-ad3b-8501e277b5df\") " Oct 30 22:15:20.317767 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.317624 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd876fc0-4344-4528-ad3b-8501e277b5df-kube-api-access-59jnv\" (UniqueName: \"kubernetes.io/projected/dd876fc0-4344-4528-ad3b-8501e277b5df-kube-api-access-59jnv\") pod \"dd876fc0-4344-4528-ad3b-8501e277b5df\" (UID: \"dd876fc0-4344-4528-ad3b-8501e277b5df\") " Oct 30 22:15:20.318024 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.317997 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd876fc0-4344-4528-ad3b-8501e277b5df-maas-api-service-ca" pod "dd876fc0-4344-4528-ad3b-8501e277b5df" (UID: "dd876fc0-4344-4528-ad3b-8501e277b5df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:20.318079 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.318065 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f959ab35-db85-43c1-882f-d6c708b3a7c7-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.318116 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.318081 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqlrm\" (UniqueName: \"kubernetes.io/projected/f959ab35-db85-43c1-882f-d6c708b3a7c7-kube-api-access-cqlrm\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.319704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.319686 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd876fc0-4344-4528-ad3b-8501e277b5df-kube-api-access-59jnv" pod "dd876fc0-4344-4528-ad3b-8501e277b5df" (UID: "dd876fc0-4344-4528-ad3b-8501e277b5df"). InnerVolumeSpecName "kube-api-access-59jnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:20.419272 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.419237 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd876fc0-4344-4528-ad3b-8501e277b5df-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.419272 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.419270 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59jnv\" (UniqueName: \"kubernetes.io/projected/dd876fc0-4344-4528-ad3b-8501e277b5df-kube-api-access-59jnv\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.869982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.862780 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" Oct 30 22:15:20.869982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.862820 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv" event={"ID":"f959ab35-db85-43c1-882f-d6c708b3a7c7","Type":"ContainerDied","Data":"f92c3ef3b04b91d9f00403811414ebdd2256c04e0576f369a0d1ee8ec3b16d8f"} Oct 30 22:15:20.869982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.862905 4859 scope.go:122] "RemoveContainer" containerID="cc199145d6ca302cff3a3ac80f5341e8722809944c4478c21ceb467be55b373a" Oct 30 22:15:20.869982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.864956 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" event={"ID":"dd876fc0-4344-4528-ad3b-8501e277b5df","Type":"ContainerDied","Data":"62fb5e28ee2aa99c11a78af9413f8e999ec17db99e6c0de331bae7ba2d971384"} Oct 30 22:15:20.869982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.865003 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr" Oct 30 22:15:20.876565 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:20.876543 4859 scope.go:122] "RemoveContainer" containerID="cce35cfc7dc8f49c401111db9a67b7bedc662ec53e70ee3b78c3f3df51edb872" Oct 30 22:15:21.030374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:21.030289 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv"] Oct 30 22:15:21.035589 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:21.035554 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889975-vbtdv"] Oct 30 22:15:21.145562 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:21.145526 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr"] Oct 30 22:15:21.149982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:21.149955 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-g52nr"] Oct 30 22:15:22.749683 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:22.749645 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" path="/var/lib/kubelet/pods/dd876fc0-4344-4528-ad3b-8501e277b5df/volumes" Oct 30 22:15:22.750295 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:22.750274 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" path="/var/lib/kubelet/pods/f959ab35-db85-43c1-882f-d6c708b3a7c7/volumes" Oct 30 22:15:25.425952 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:25.425910 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:15:39.876795 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:15:39.876759 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:15:47.099120 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.099086 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv"] Oct 30 22:15:47.101030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101004 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerName="cleanup" Oct 30 22:15:47.101030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101031 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerName="cleanup" Oct 30 22:15:47.101177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101079 4859 cpu_manager.go:414] "containerMap: removing container" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerName="cleanup" Oct 30 22:15:47.101177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101087 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerName="cleanup" Oct 30 22:15:47.101177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101099 4859 cpu_manager.go:414] "containerMap: removing container" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerName="cleanup" Oct 30 22:15:47.101177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101106 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerName="cleanup" Oct 30 22:15:47.101177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101122 4859 cpu_manager.go:414] "containerMap: removing container" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerName="cleanup" Oct 30 22:15:47.101177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101129 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerName="cleanup" Oct 30 22:15:47.101177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101148 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerName="cleanup" Oct 30 22:15:47.101177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101155 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerName="cleanup" Oct 30 22:15:47.101461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101309 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerName="cleanup" Oct 30 22:15:47.101461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101324 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerName="cleanup" Oct 30 22:15:47.101461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101336 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerName="cleanup" Oct 30 22:15:47.101461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101346 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd876fc0-4344-4528-ad3b-8501e277b5df" containerName="cleanup" Oct 30 22:15:47.101461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101358 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerName="cleanup" Oct 30 22:15:47.101461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.101369 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerName="cleanup" Oct 30 22:15:47.106143 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.106111 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.177270 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.177229 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.177547 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.177517 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98bbb81-f4dc-493d-92c7-d43984c30517-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.177667 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.177564 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hts8n\" (UniqueName: \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-kube-api-access-hts8n\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.177832 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.177809 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.177907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.177886 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.177988 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.177971 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.178064 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.178040 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.178107 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.178086 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.178152 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.178124 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.225381 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.225351 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv"] Oct 30 22:15:47.279225 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279186 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.279390 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279283 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.279390 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279314 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.279390 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279344 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.279549 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279387 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.279549 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279445 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.279549 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279483 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98bbb81-f4dc-493d-92c7-d43984c30517-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.279549 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279517 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hts8n\" (UniqueName: \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-kube-api-access-hts8n\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.279754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279669 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.279820 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279791 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.279982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.279934 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.280165 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.280143 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.280240 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.280167 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.280285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.280267 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98bbb81-f4dc-493d-92c7-d43984c30517-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.281832 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.281803 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.281984 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.281960 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.337704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.337676 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hts8n\" (UniqueName: \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-kube-api-access-hts8n\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.337816 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.337681 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4bfdv\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.424290 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.424205 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:47.854692 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.854656 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv"] Oct 30 22:15:47.857222 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:15:47.857183 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98bbb81_f4dc_493d_92c7_d43984c30517.slice/crio-4665dcef37718ba00e0f39a5b22b52971f6fb4ab5c6af2127fd58f32e7a0f9e9 WatchSource:0}: Error finding container 4665dcef37718ba00e0f39a5b22b52971f6fb4ab5c6af2127fd58f32e7a0f9e9: Status 404 returned error can't find the container with id 4665dcef37718ba00e0f39a5b22b52971f6fb4ab5c6af2127fd58f32e7a0f9e9 Oct 30 22:15:47.860370 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.860335 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:47.860459 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.860404 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:47.860459 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:47.860430 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:48.109624 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:48.109536 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" event={"ID":"c98bbb81-f4dc-493d-92c7-d43984c30517","Type":"ContainerStarted","Data":"55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a"} Oct 30 22:15:48.109624 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:48.109600 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" event={"ID":"c98bbb81-f4dc-493d-92c7-d43984c30517","Type":"ContainerStarted","Data":"4665dcef37718ba00e0f39a5b22b52971f6fb4ab5c6af2127fd58f32e7a0f9e9"} Oct 30 22:15:48.229788 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:48.229720 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podStartSLOduration=1.229700588 podStartE2EDuration="1.229700588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:48.227752326 +0000 UTC m=+2005.600379075" watchObservedRunningTime="2026-10-30 22:15:48.229700588 +0000 UTC m=+2005.602327336" Oct 30 22:15:48.426281 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:48.425501 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:48.427009 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:48.426975 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4bfdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:15:48.427240 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:48.427209 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:15:49.424646 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:49.424608 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4bfdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:15:49.425194 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:49.424671 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:15:50.424834 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:50.424797 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4bfdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:15:50.425248 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:50.424853 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:15:51.425430 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:51.425393 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4bfdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:15:51.425813 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:51.425449 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:15:52.424827 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:52.424790 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4bfdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:15:52.425105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:52.424849 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:15:53.426303 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:53.426268 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4bfdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:15:53.426708 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:53.426320 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:15:54.425181 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:54.425139 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4bfdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:15:54.425356 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:54.425205 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:15:55.424596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:55.424557 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4bfdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:15:55.425010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:55.424620 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:15:56.434310 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:56.434261 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:56.435112 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:56.435080 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:15:56.436379 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:15:56.436354 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:16:02.212058 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.212025 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m"] Oct 30 22:16:02.212625 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.212611 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerName="cleanup" Oct 30 22:16:02.212669 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.212626 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f959ab35-db85-43c1-882f-d6c708b3a7c7" containerName="cleanup" Oct 30 22:16:02.216388 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.216369 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.399908 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.399874 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.400146 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.399920 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.400146 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.399940 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.400146 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.400085 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.400146 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.400130 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.400375 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.400301 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571531de-dec4-474a-b197-8d84823a8198-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.400435 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.400413 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwd46\" (UniqueName: \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-kube-api-access-mwd46\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.400472 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.400450 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.400522 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.400479 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571531de-dec4-474a-b197-8d84823a8198-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.418075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.418039 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m"] Oct 30 22:16:02.501760 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.501666 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwd46\" (UniqueName: \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-kube-api-access-mwd46\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.501760 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.501711 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.501760 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.501744 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571531de-dec4-474a-b197-8d84823a8198-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.501802 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.501841 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.501871 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.502026 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.502095 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.502216 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571531de-dec4-474a-b197-8d84823a8198-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.502260 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.502285 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.502337 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502571 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.502547 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.502615 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.502599 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571531de-dec4-474a-b197-8d84823a8198-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.504215 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.504192 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.504541 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.504524 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571531de-dec4-474a-b197-8d84823a8198-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.558412 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.558377 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.558710 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.558687 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwd46\" (UniqueName: \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-kube-api-access-mwd46\") pod \"maas-default-gateway-openshift-default-694db55c87-msm7m\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:02.830641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:02.830553 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:03.257192 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:03.257158 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m"] Oct 30 22:16:03.258113 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:16:03.258085 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571531de_dec4_474a_b197_8d84823a8198.slice/crio-ef7ec32908470f315324e96daa352028a6fbd2db7a5460a7bd81cb1dcc30b303 WatchSource:0}: Error finding container ef7ec32908470f315324e96daa352028a6fbd2db7a5460a7bd81cb1dcc30b303: Status 404 returned error can't find the container with id ef7ec32908470f315324e96daa352028a6fbd2db7a5460a7bd81cb1dcc30b303 Oct 30 22:16:03.260782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:03.260747 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:16:03.260861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:03.260814 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:16:03.260861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:03.260840 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:16:04.225917 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:04.225879 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" event={"ID":"571531de-dec4-474a-b197-8d84823a8198","Type":"ContainerStarted","Data":"4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f"} Oct 30 22:16:04.226157 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:04.225935 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" event={"ID":"571531de-dec4-474a-b197-8d84823a8198","Type":"ContainerStarted","Data":"ef7ec32908470f315324e96daa352028a6fbd2db7a5460a7bd81cb1dcc30b303"} Oct 30 22:16:04.449256 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:04.448028 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" podStartSLOduration=2.4480099109999998 podStartE2EDuration="2.448009911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:04.393541499 +0000 UTC m=+2021.766168246" watchObservedRunningTime="2026-10-30 22:16:04.448009911 +0000 UTC m=+2021.820636659" Oct 30 22:16:04.830738 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:04.830706 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:04.832240 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:04.832213 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:16:04.832361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:04.832267 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:16:05.831486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:05.831450 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:16:05.831882 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:05.831506 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:16:06.831850 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:06.831812 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:16:06.832280 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:06.831868 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:16:07.690360 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.690327 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6"] Oct 30 22:16:07.694470 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.694448 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.787112 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.787073 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-metrics-certs\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.787112 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.787120 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.787408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.787256 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454daf2-9772-4eba-81fc-d9acf70939ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.787408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.787383 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82x9x\" (UniqueName: \"kubernetes.io/projected/f454daf2-9772-4eba-81fc-d9acf70939ea-kube-api-access-82x9x\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.798881 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.798844 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2edb49-serving-cert\"" Oct 30 22:16:07.802628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.802598 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6"] Oct 30 22:16:07.831559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.831523 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:16:07.831728 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.831585 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:16:07.888206 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.888172 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82x9x\" (UniqueName: \"kubernetes.io/projected/f454daf2-9772-4eba-81fc-d9acf70939ea-kube-api-access-82x9x\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.888695 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.888338 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-metrics-certs\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.888695 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.888377 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.888695 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.888442 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454daf2-9772-4eba-81fc-d9acf70939ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.888695 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:16:07.888498 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-serving-cert: secret "maas-api-e2e-ipp-a-2edb49-serving-cert" not found Oct 30 22:16:07.888695 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:16:07.888555 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-maas-api-tls podName:f454daf2-9772-4eba-81fc-d9acf70939ea nodeName:}" failed. No retries permitted until 2026-10-30 22:16:08.388542624 +0000 UTC m=+2025.761169350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-maas-api-tls") pod "maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" (UID: "f454daf2-9772-4eba-81fc-d9acf70939ea") : secret "maas-api-e2e-ipp-a-2edb49-serving-cert" not found Oct 30 22:16:07.889211 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.889188 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454daf2-9772-4eba-81fc-d9acf70939ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.890776 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.890756 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-metrics-certs\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:07.948152 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:07.948079 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82x9x\" (UniqueName: \"kubernetes.io/projected/f454daf2-9772-4eba-81fc-d9acf70939ea-kube-api-access-82x9x\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:08.399223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:08.399162 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:08.401726 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:08.401692 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:08.608703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:08.608667 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:08.830869 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:08.830832 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:16:08.831063 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:08.830893 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:16:09.012333 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:09.012303 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6"] Oct 30 22:16:09.014584 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:16:09.014542 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf454daf2_9772_4eba_81fc_d9acf70939ea.slice/crio-f27db2a325a6013cb8da239d8296df53e73adc23b75a2b8451eca7700317cc54 WatchSource:0}: Error finding container f27db2a325a6013cb8da239d8296df53e73adc23b75a2b8451eca7700317cc54: Status 404 returned error can't find the container with id f27db2a325a6013cb8da239d8296df53e73adc23b75a2b8451eca7700317cc54 Oct 30 22:16:09.276790 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:09.276691 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" event={"ID":"f454daf2-9772-4eba-81fc-d9acf70939ea","Type":"ContainerStarted","Data":"f27db2a325a6013cb8da239d8296df53e73adc23b75a2b8451eca7700317cc54"} Oct 30 22:16:09.831532 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:09.831494 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:16:09.831713 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:09.831561 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:16:09.882408 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:16:09.882369 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:16:10.285061 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:10.285016 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" event={"ID":"f454daf2-9772-4eba-81fc-d9acf70939ea","Type":"ContainerStarted","Data":"72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39"} Oct 30 22:16:10.285444 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:10.285087 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:10.402161 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:10.402106 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" podStartSLOduration=2.516979711 podStartE2EDuration="3.402092939s" totalImagesPullingTime="885.113228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:10.399707222 +0000 UTC m=+2027.772333972" watchObservedRunningTime="2026-10-30 22:16:10.402092939 +0000 UTC m=+2027.774719687" Oct 30 22:16:10.831474 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:10.831433 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:16:10.831677 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:10.831514 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:16:11.947021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:11.946985 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:11.947711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:11.947681 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:11.948361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:11.948338 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:16:16.301086 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:16.301042 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:16:24.050970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.044000 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976"] Oct 30 22:16:24.085967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.078038 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" podUID="9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74" containerName="maas-api" containerID="cri-o://553a4bf35cdeff921ca39a9f3022c832a72eb115e55f2955581286ac4190cd80" gracePeriod=30 Oct 30 22:16:24.275135 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.274008 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:16:24.395359 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.395222 4859 generic.go:356] "Generic (PLEG): container finished" podID="9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74" containerID="553a4bf35cdeff921ca39a9f3022c832a72eb115e55f2955581286ac4190cd80" exitCode=0 Oct 30 22:16:24.395359 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.395292 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" event={"ID":"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74","Type":"ContainerDied","Data":"553a4bf35cdeff921ca39a9f3022c832a72eb115e55f2955581286ac4190cd80"} Oct 30 22:16:24.500852 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.500826 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:16:24.654895 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.654799 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-kube-api-access-7d6pf\" (UniqueName: \"kubernetes.io/projected/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-kube-api-access-7d6pf\") pod \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " Oct 30 22:16:24.655084 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.654930 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-metrics-certs\") pod \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " Oct 30 22:16:24.655084 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.655066 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-maas-api-tls\") pod \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " Oct 30 22:16:24.655218 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.655120 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-openshift-service-ca\") pod \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\" (UID: \"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74\") " Oct 30 22:16:24.656018 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.655989 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-openshift-service-ca" pod "9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74" (UID: "9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:24.662511 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.662480 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-metrics-certs" pod "9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74" (UID: "9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:24.662676 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.662505 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-kube-api-access-7d6pf" pod "9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74" (UID: "9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74"). InnerVolumeSpecName "kube-api-access-7d6pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:24.664339 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.664309 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-maas-api-tls" pod "9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74" (UID: "9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:24.769354 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.768884 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d6pf\" (UniqueName: \"kubernetes.io/projected/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-kube-api-access-7d6pf\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:24.769354 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.768931 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:24.769354 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.768958 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:24.769354 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:24.768972 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:25.405350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:25.405189 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" event={"ID":"9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74","Type":"ContainerDied","Data":"79f8137217ba5fac8c8da05cc107806077dae5f1baaf9bad927f45b59fd68c9a"} Oct 30 22:16:25.405350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:25.405265 4859 scope.go:122] "RemoveContainer" containerID="553a4bf35cdeff921ca39a9f3022c832a72eb115e55f2955581286ac4190cd80" Oct 30 22:16:25.405891 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:25.405439 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976" Oct 30 22:16:25.606923 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:25.606884 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976"] Oct 30 22:16:25.656138 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:25.656052 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97ca39-85dd7675fc-h5976"] Oct 30 22:16:26.742082 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:26.742038 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74" path="/var/lib/kubelet/pods/9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74/volumes" Oct 30 22:16:30.790970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.785036 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8"] Oct 30 22:16:30.790970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.785933 4859 cpu_manager.go:414] "containerMap: removing container" podUID="9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74" containerName="maas-api" Oct 30 22:16:30.790970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.785967 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74" containerName="maas-api" Oct 30 22:16:30.790970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.786171 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4aee3c-d4e7-47ef-9e8f-1dd2537ddb74" containerName="maas-api" Oct 30 22:16:30.796314 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.796288 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:30.832661 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.832629 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8"] Oct 30 22:16:30.889968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.889840 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-577ea8-serving-cert\"" Oct 30 22:16:30.919434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.919336 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:30.919673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.919446 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-metrics-certs\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:30.919673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.919523 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:30.919673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:30.919662 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpjpx\" (UniqueName: \"kubernetes.io/projected/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-kube-api-access-xpjpx\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:31.020909 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:31.020868 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:31.021269 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:31.021249 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-metrics-certs\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:31.021469 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:31.021442 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:31.021766 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:31.021726 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpjpx\" (UniqueName: \"kubernetes.io/projected/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-kube-api-access-xpjpx\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:31.021903 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:16:31.021288 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-serving-cert: secret "maas-api-e2e-ipp-b-577ea8-serving-cert" not found Oct 30 22:16:31.022086 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:16:31.022070 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-maas-api-tls podName:89c5c684-7187-4ec6-b01a-fb81ef01f7ab nodeName:}" failed. No retries permitted until 2026-10-30 22:16:31.522045603 +0000 UTC m=+2048.894672340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-maas-api-tls") pod "maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" (UID: "89c5c684-7187-4ec6-b01a-fb81ef01f7ab") : secret "maas-api-e2e-ipp-b-577ea8-serving-cert" not found Oct 30 22:16:31.022512 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:31.022485 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:31.027663 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:31.025411 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-metrics-certs\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:31.107513 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:31.107350 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpjpx\" (UniqueName: \"kubernetes.io/projected/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-kube-api-access-xpjpx\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:31.528995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:31.528922 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:31.531598 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:31.531568 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:31.711412 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:31.711366 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:32.098365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:32.098338 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8"] Oct 30 22:16:32.101262 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:16:32.101229 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c5c684_7187_4ec6_b01a_fb81ef01f7ab.slice/crio-1d796c46c65ec45edb009bb74d372965988e3d7168ff7d6fe3476f23196dcb24 WatchSource:0}: Error finding container 1d796c46c65ec45edb009bb74d372965988e3d7168ff7d6fe3476f23196dcb24: Status 404 returned error can't find the container with id 1d796c46c65ec45edb009bb74d372965988e3d7168ff7d6fe3476f23196dcb24 Oct 30 22:16:32.457255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:32.457172 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" event={"ID":"89c5c684-7187-4ec6-b01a-fb81ef01f7ab","Type":"ContainerStarted","Data":"1d796c46c65ec45edb009bb74d372965988e3d7168ff7d6fe3476f23196dcb24"} Oct 30 22:16:33.485590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:33.484368 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" event={"ID":"89c5c684-7187-4ec6-b01a-fb81ef01f7ab","Type":"ContainerStarted","Data":"8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8"} Oct 30 22:16:33.485590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:33.484957 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:33.604199 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:33.604134 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" podStartSLOduration=2.7647763039999997 podStartE2EDuration="3.604113258s" totalImagesPullingTime="839.336954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:33.600913189 +0000 UTC m=+2050.973539937" watchObservedRunningTime="2026-10-30 22:16:33.604113258 +0000 UTC m=+2050.976740006" Oct 30 22:16:35.501883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:35.501798 4859 generic.go:356] "Generic (PLEG): container finished" podID="9d60d248-a783-420e-a309-16db1b7ee00e" containerID="a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7" exitCode=0 Oct 30 22:16:35.501883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:35.501868 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" event={"ID":"9d60d248-a783-420e-a309-16db1b7ee00e","Type":"ContainerDied","Data":"a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7"} Oct 30 22:16:35.502343 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:35.502251 4859 scope.go:122] "RemoveContainer" containerID="a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7" Oct 30 22:16:35.503687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:35.503651 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:16:35.503767 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:35.503747 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:16:35.503805 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:35.503793 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:16:36.510756 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:36.510716 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" event={"ID":"9d60d248-a783-420e-a309-16db1b7ee00e","Type":"ContainerStarted","Data":"781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8"} Oct 30 22:16:37.282967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:37.282377 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:16:37.282967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:37.282507 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:16:37.282967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:37.282554 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 22:16:38.279969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:38.279118 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:16:38.279969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:38.279202 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 22:16:39.281968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:39.272316 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:16:39.281968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:39.272376 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 22:16:39.500553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:39.500523 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:16:39.904176 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:16:39.904137 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:16:40.274022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:40.273376 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:16:40.274022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:40.273437 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 22:16:41.280968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:41.272389 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:16:41.280968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:41.272447 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 22:16:42.274854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:42.274820 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:16:42.275210 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:42.275183 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:16:42.276048 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:42.276031 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:16:51.351884 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:16:51.351802 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:17:09.878509 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:17:09.878470 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:17:23.133419 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:17:23.133385 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:17:23.151644 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:17:23.148351 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:17:23.153883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:17:23.153855 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:17:23.162346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:17:23.162315 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:17:33.966433 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:17:33.966395 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:17:39.876324 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:17:39.876242 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:18:09.911188 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:18:09.911143 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:18:16.486387 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:16.486343 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:18:23.678920 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.678883 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz"] Oct 30 22:18:23.684358 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.684325 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:23.738580 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.738548 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz"] Oct 30 22:18:23.783567 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.783525 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-96567a-serving-cert\"" Oct 30 22:18:23.839065 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.839034 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-metrics-certs\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:23.839065 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.839067 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:23.839326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.839297 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhcjx\" (UniqueName: \"kubernetes.io/projected/c8a42bc5-a366-43fa-b540-3ad76aee16bb-kube-api-access-bhcjx\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:23.839415 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.839395 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a42bc5-a366-43fa-b540-3ad76aee16bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:23.943244 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.943102 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhcjx\" (UniqueName: \"kubernetes.io/projected/c8a42bc5-a366-43fa-b540-3ad76aee16bb-kube-api-access-bhcjx\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:23.943244 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.943182 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a42bc5-a366-43fa-b540-3ad76aee16bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:23.943502 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.943327 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-metrics-certs\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:23.943502 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.943359 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:23.943622 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:18:23.943524 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-serving-cert: secret "maas-api-e2e-ipp-b-96567a-serving-cert" not found Oct 30 22:18:23.943622 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:18:23.943589 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-maas-api-tls podName:c8a42bc5-a366-43fa-b540-3ad76aee16bb nodeName:}" failed. No retries permitted until 2026-10-30 22:18:24.443569418 +0000 UTC m=+2161.816196161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-maas-api-tls") pod "maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" (UID: "c8a42bc5-a366-43fa-b540-3ad76aee16bb") : secret "maas-api-e2e-ipp-b-96567a-serving-cert" not found Oct 30 22:18:23.944890 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.944748 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a42bc5-a366-43fa-b540-3ad76aee16bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:23.947542 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:23.947487 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-metrics-certs\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:24.002981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:24.002936 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhcjx\" (UniqueName: \"kubernetes.io/projected/c8a42bc5-a366-43fa-b540-3ad76aee16bb-kube-api-access-bhcjx\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:24.447965 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:24.447913 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:24.450319 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:24.450297 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:24.601108 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:24.601066 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:24.982667 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:24.982615 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz"] Oct 30 22:18:24.984595 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:18:24.984553 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a42bc5_a366_43fa_b540_3ad76aee16bb.slice/crio-739f3305e0a9e433fd5693c46573a56d9e56d32b5f1f810df1c1fb0ff5e3b5c5 WatchSource:0}: Error finding container 739f3305e0a9e433fd5693c46573a56d9e56d32b5f1f810df1c1fb0ff5e3b5c5: Status 404 returned error can't find the container with id 739f3305e0a9e433fd5693c46573a56d9e56d32b5f1f810df1c1fb0ff5e3b5c5 Oct 30 22:18:24.985926 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:24.985903 4859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:18:25.440734 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:25.440695 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" event={"ID":"c8a42bc5-a366-43fa-b540-3ad76aee16bb","Type":"ContainerStarted","Data":"739f3305e0a9e433fd5693c46573a56d9e56d32b5f1f810df1c1fb0ff5e3b5c5"} Oct 30 22:18:26.451756 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:26.451722 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" event={"ID":"c8a42bc5-a366-43fa-b540-3ad76aee16bb","Type":"ContainerStarted","Data":"e1117c30da071cacf4b4623353c41d7f752fde2181568886a6d03881ebdd2b13"} Oct 30 22:18:26.452263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:26.451936 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:26.566683 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:26.566617 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" podStartSLOduration=2.611025794 podStartE2EDuration="3.56658576s" totalImagesPullingTime="955.559966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:26.565083737 +0000 UTC m=+2163.937710495" watchObservedRunningTime="2026-10-30 22:18:26.56658576 +0000 UTC m=+2163.939212507" Oct 30 22:18:32.464638 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:32.464605 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:18:39.880462 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:18:39.880386 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:18:45.043661 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:18:45.043622 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:19:09.394212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:09.394130 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b"] Oct 30 22:19:09.394658 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:09.394358 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" containerID="cri-o://d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf" gracePeriod=30 Oct 30 22:19:09.877773 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:19:09.877737 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:19:10.331408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:10.331320 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg"] Oct 30 22:19:10.331567 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:10.331535 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" containerID="cri-o://56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33" gracePeriod=30 Oct 30 22:19:10.953044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:10.953004 4859 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:19:10.953419 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:10.953068 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:19:13.264292 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:13.264255 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:19:13.264660 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:13.264307 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:19:14.425471 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:19:14.425440 4859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef6c3cb_d4e0_4770_8a85_919938bd155a.slice/crio-d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:19:14.669620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.669555 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:19:14.694025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.693987 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-podinfo\") pod \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " Oct 30 22:19:14.694177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694110 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-token\" (UniqueName: \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-token\") pod \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " Oct 30 22:19:14.694177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694153 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-kube-api-access-vfn7m\" (UniqueName: \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-kube-api-access-vfn7m\") pod \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " Oct 30 22:19:14.694291 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694209 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-credential-socket\") pod \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " Oct 30 22:19:14.694291 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694236 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-data\") pod \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " Oct 30 22:19:14.694400 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694299 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-socket\") pod \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " Oct 30 22:19:14.694400 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694324 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-certs\") pod \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " Oct 30 22:19:14.694400 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694371 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istiod-ca-cert\") pod \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " Oct 30 22:19:14.694549 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694401 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-envoy\") pod \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\" (UID: \"8ef6c3cb-d4e0-4770-8a85-919938bd155a\") " Oct 30 22:19:14.694613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694545 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-credential-socket" pod "8ef6c3cb-d4e0-4770-8a85-919938bd155a" (UID: "8ef6c3cb-d4e0-4770-8a85-919938bd155a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:14.694613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694593 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-data" pod "8ef6c3cb-d4e0-4770-8a85-919938bd155a" (UID: "8ef6c3cb-d4e0-4770-8a85-919938bd155a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:14.694721 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694628 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-socket" pod "8ef6c3cb-d4e0-4770-8a85-919938bd155a" (UID: "8ef6c3cb-d4e0-4770-8a85-919938bd155a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:14.694911 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694859 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-certs" pod "8ef6c3cb-d4e0-4770-8a85-919938bd155a" (UID: "8ef6c3cb-d4e0-4770-8a85-919938bd155a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:14.695064 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.694988 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istiod-ca-cert" pod "8ef6c3cb-d4e0-4770-8a85-919938bd155a" (UID: "8ef6c3cb-d4e0-4770-8a85-919938bd155a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:14.695064 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.695056 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:14.695193 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.695079 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:14.695193 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.695094 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:14.695193 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.695107 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:14.696262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.696240 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-podinfo" pod "8ef6c3cb-d4e0-4770-8a85-919938bd155a" (UID: "8ef6c3cb-d4e0-4770-8a85-919938bd155a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:14.696420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.696402 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-token" pod "8ef6c3cb-d4e0-4770-8a85-919938bd155a" (UID: "8ef6c3cb-d4e0-4770-8a85-919938bd155a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:14.696820 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.696798 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-envoy" pod "8ef6c3cb-d4e0-4770-8a85-919938bd155a" (UID: "8ef6c3cb-d4e0-4770-8a85-919938bd155a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:14.696820 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.696805 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-kube-api-access-vfn7m" pod "8ef6c3cb-d4e0-4770-8a85-919938bd155a" (UID: "8ef6c3cb-d4e0-4770-8a85-919938bd155a"). InnerVolumeSpecName "kube-api-access-vfn7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:14.797118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.796885 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:14.797307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.797125 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:14.797307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.797145 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:14.797307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.797160 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:14.797307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.797177 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfn7m\" (UniqueName: \"kubernetes.io/projected/8ef6c3cb-d4e0-4770-8a85-919938bd155a-kube-api-access-vfn7m\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:14.798161 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.798125 4859 generic.go:356] "Generic (PLEG): container finished" podID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerID="d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf" exitCode=0 Oct 30 22:19:14.798300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.798174 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" Oct 30 22:19:14.798300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.798203 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" event={"ID":"8ef6c3cb-d4e0-4770-8a85-919938bd155a","Type":"ContainerDied","Data":"d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf"} Oct 30 22:19:14.798300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.798259 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b" event={"ID":"8ef6c3cb-d4e0-4770-8a85-919938bd155a","Type":"ContainerDied","Data":"5f2b552a73037659099575120fd10cc0b6363c363508ba4b6002be37c0f2b0f0"} Oct 30 22:19:14.798300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.798282 4859 scope.go:122] "RemoveContainer" containerID="d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf" Oct 30 22:19:14.808754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.808732 4859 scope.go:122] "RemoveContainer" containerID="d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf" Oct 30 22:19:14.809085 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:19:14.809064 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf\": container with ID starting with d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf not found: ID does not exist" containerID="d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf" Oct 30 22:19:14.809179 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.809103 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf"} err="failed to get container status \"d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf\": rpc error: code = NotFound desc = could not find container \"d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf\": container with ID starting with d439b9bbe7950d8ccdf905d3171522685c48c8b71bc0300295b5b668b4f2b3cf not found: ID does not exist" Oct 30 22:19:14.950508 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.950421 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b"] Oct 30 22:19:14.953642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:14.953619 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cbl2b"] Oct 30 22:19:15.599405 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.599340 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:19:15.707454 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.707410 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-credential-socket\") pod \"847c5aab-e0f2-4d3f-8484-77549e797e75\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " Oct 30 22:19:15.707646 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.707477 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-envoy\") pod \"847c5aab-e0f2-4d3f-8484-77549e797e75\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " Oct 30 22:19:15.707646 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.707538 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-data\") pod \"847c5aab-e0f2-4d3f-8484-77549e797e75\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " Oct 30 22:19:15.707646 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.707619 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-kube-api-access-mbmnt\" (UniqueName: \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-kube-api-access-mbmnt\") pod \"847c5aab-e0f2-4d3f-8484-77549e797e75\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " Oct 30 22:19:15.707818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.707675 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847c5aab-e0f2-4d3f-8484-77549e797e75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847c5aab-e0f2-4d3f-8484-77549e797e75-istiod-ca-cert\") pod \"847c5aab-e0f2-4d3f-8484-77549e797e75\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " Oct 30 22:19:15.707818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.707702 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-token\" (UniqueName: \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-token\") pod \"847c5aab-e0f2-4d3f-8484-77549e797e75\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " Oct 30 22:19:15.707818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.707700 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-credential-socket" pod "847c5aab-e0f2-4d3f-8484-77549e797e75" (UID: "847c5aab-e0f2-4d3f-8484-77549e797e75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:15.708367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.707894 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-socket\") pod \"847c5aab-e0f2-4d3f-8484-77549e797e75\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " Oct 30 22:19:15.708507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.707929 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-data" pod "847c5aab-e0f2-4d3f-8484-77549e797e75" (UID: "847c5aab-e0f2-4d3f-8484-77549e797e75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:15.708507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.708122 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-socket" pod "847c5aab-e0f2-4d3f-8484-77549e797e75" (UID: "847c5aab-e0f2-4d3f-8484-77549e797e75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:15.708507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.708160 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847c5aab-e0f2-4d3f-8484-77549e797e75-istiod-ca-cert" pod "847c5aab-e0f2-4d3f-8484-77549e797e75" (UID: "847c5aab-e0f2-4d3f-8484-77549e797e75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:15.708507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.708483 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-certs\") pod \"847c5aab-e0f2-4d3f-8484-77549e797e75\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " Oct 30 22:19:15.708766 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.708746 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-certs" pod "847c5aab-e0f2-4d3f-8484-77549e797e75" (UID: "847c5aab-e0f2-4d3f-8484-77549e797e75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:15.708831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.708817 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-podinfo\") pod \"847c5aab-e0f2-4d3f-8484-77549e797e75\" (UID: \"847c5aab-e0f2-4d3f-8484-77549e797e75\") " Oct 30 22:19:15.709398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.709377 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:15.709517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.709400 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847c5aab-e0f2-4d3f-8484-77549e797e75-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:15.709517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.709414 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:15.709517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.709427 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:15.709517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.709440 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:15.710278 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.710251 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-envoy" pod "847c5aab-e0f2-4d3f-8484-77549e797e75" (UID: "847c5aab-e0f2-4d3f-8484-77549e797e75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:15.710382 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.710252 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-kube-api-access-mbmnt" pod "847c5aab-e0f2-4d3f-8484-77549e797e75" (UID: "847c5aab-e0f2-4d3f-8484-77549e797e75"). InnerVolumeSpecName "kube-api-access-mbmnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:15.710599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.710576 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-token" pod "847c5aab-e0f2-4d3f-8484-77549e797e75" (UID: "847c5aab-e0f2-4d3f-8484-77549e797e75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:15.711313 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.711294 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-podinfo" pod "847c5aab-e0f2-4d3f-8484-77549e797e75" (UID: "847c5aab-e0f2-4d3f-8484-77549e797e75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:15.809353 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.809317 4859 generic.go:356] "Generic (PLEG): container finished" podID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerID="56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33" exitCode=0 Oct 30 22:19:15.809632 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.809364 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" Oct 30 22:19:15.809632 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.809396 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" event={"ID":"847c5aab-e0f2-4d3f-8484-77549e797e75","Type":"ContainerDied","Data":"56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33"} Oct 30 22:19:15.809632 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.809460 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg" event={"ID":"847c5aab-e0f2-4d3f-8484-77549e797e75","Type":"ContainerDied","Data":"81e7e3e9a3ee3b0b811ea41e59e06fe96c7bca956884918058c05665d8d69525"} Oct 30 22:19:15.809632 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.809490 4859 scope.go:122] "RemoveContainer" containerID="56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33" Oct 30 22:19:15.810467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.810437 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbmnt\" (UniqueName: \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-kube-api-access-mbmnt\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:15.810594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.810470 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:15.810594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.810485 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:15.810594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.810499 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847c5aab-e0f2-4d3f-8484-77549e797e75-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:15.820807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.820788 4859 scope.go:122] "RemoveContainer" containerID="56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33" Oct 30 22:19:15.821096 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:19:15.821076 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33\": container with ID starting with 56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33 not found: ID does not exist" containerID="56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33" Oct 30 22:19:15.821170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.821109 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33"} err="failed to get container status \"56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33\": rpc error: code = NotFound desc = could not find container \"56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33\": container with ID starting with 56e5ca76e309999bb27f6d8f319181802a43b1e938e1672d79c602fad6ff2a33 not found: ID does not exist" Oct 30 22:19:15.981326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.981233 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg"] Oct 30 22:19:15.984084 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:15.984058 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-dmltg"] Oct 30 22:19:16.741317 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:16.741284 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" path="/var/lib/kubelet/pods/847c5aab-e0f2-4d3f-8484-77549e797e75/volumes" Oct 30 22:19:16.741761 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:16.741747 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" path="/var/lib/kubelet/pods/8ef6c3cb-d4e0-4770-8a85-919938bd155a/volumes" Oct 30 22:19:30.503368 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.503328 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh"] Oct 30 22:19:30.503824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.503559 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" podUID="df95d869-36df-46ea-861c-7a01b29cf84f" containerName="maas-api" containerID="cri-o://0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2" gracePeriod=30 Oct 30 22:19:30.803978 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.803931 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:19:30.904565 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.904532 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df95d869-36df-46ea-861c-7a01b29cf84f-kube-api-access-kpjbq\" (UniqueName: \"kubernetes.io/projected/df95d869-36df-46ea-861c-7a01b29cf84f-kube-api-access-kpjbq\") pod \"df95d869-36df-46ea-861c-7a01b29cf84f\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " Oct 30 22:19:30.904735 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.904617 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df95d869-36df-46ea-861c-7a01b29cf84f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df95d869-36df-46ea-861c-7a01b29cf84f-openshift-service-ca\") pod \"df95d869-36df-46ea-861c-7a01b29cf84f\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " Oct 30 22:19:30.904776 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.904735 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-metrics-certs\") pod \"df95d869-36df-46ea-861c-7a01b29cf84f\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " Oct 30 22:19:30.904823 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.904772 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-maas-api-tls\") pod \"df95d869-36df-46ea-861c-7a01b29cf84f\" (UID: \"df95d869-36df-46ea-861c-7a01b29cf84f\") " Oct 30 22:19:30.905060 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.905032 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df95d869-36df-46ea-861c-7a01b29cf84f-openshift-service-ca" pod "df95d869-36df-46ea-861c-7a01b29cf84f" (UID: "df95d869-36df-46ea-861c-7a01b29cf84f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:30.905448 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.905407 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df95d869-36df-46ea-861c-7a01b29cf84f-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:30.906799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.906777 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df95d869-36df-46ea-861c-7a01b29cf84f-kube-api-access-kpjbq" pod "df95d869-36df-46ea-861c-7a01b29cf84f" (UID: "df95d869-36df-46ea-861c-7a01b29cf84f"). InnerVolumeSpecName "kube-api-access-kpjbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:30.907226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.907197 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-metrics-certs" pod "df95d869-36df-46ea-861c-7a01b29cf84f" (UID: "df95d869-36df-46ea-861c-7a01b29cf84f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:30.907313 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.907250 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-maas-api-tls" pod "df95d869-36df-46ea-861c-7a01b29cf84f" (UID: "df95d869-36df-46ea-861c-7a01b29cf84f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:30.915034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.915009 4859 generic.go:356] "Generic (PLEG): container finished" podID="df95d869-36df-46ea-861c-7a01b29cf84f" containerID="0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2" exitCode=0 Oct 30 22:19:30.915168 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.915142 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" Oct 30 22:19:30.915245 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.915142 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" event={"ID":"df95d869-36df-46ea-861c-7a01b29cf84f","Type":"ContainerDied","Data":"0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2"} Oct 30 22:19:30.915290 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.915280 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh" event={"ID":"df95d869-36df-46ea-861c-7a01b29cf84f","Type":"ContainerDied","Data":"a9d1bace1367b2d5b4b7bfc97e8b98a6929b7d0e2800626f67ba6be1742a5d9e"} Oct 30 22:19:30.915323 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.915306 4859 scope.go:122] "RemoveContainer" containerID="0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2" Oct 30 22:19:30.939051 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.939025 4859 scope.go:122] "RemoveContainer" containerID="0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2" Oct 30 22:19:30.939415 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:19:30.939394 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2\": container with ID starting with 0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2 not found: ID does not exist" containerID="0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2" Oct 30 22:19:30.939503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:30.939423 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2"} err="failed to get container status \"0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2\": rpc error: code = NotFound desc = could not find container \"0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2\": container with ID starting with 0b7da787ea9fb596b3847ac6243037704c73a0fb5b241fe620edeb4de081bdf2 not found: ID does not exist" Oct 30 22:19:31.007022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:31.006986 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpjbq\" (UniqueName: \"kubernetes.io/projected/df95d869-36df-46ea-861c-7a01b29cf84f-kube-api-access-kpjbq\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:31.007022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:31.007016 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:31.007022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:31.007028 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df95d869-36df-46ea-861c-7a01b29cf84f-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:31.084198 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:31.084109 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh"] Oct 30 22:19:31.089643 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:31.089611 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7a54db-f65fbcbc4-gj4qh"] Oct 30 22:19:32.741226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:32.741190 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df95d869-36df-46ea-861c-7a01b29cf84f" path="/var/lib/kubelet/pods/df95d869-36df-46ea-861c-7a01b29cf84f/volumes" Oct 30 22:19:39.896665 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:19:39.896620 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:19:42.707971 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:42.707661 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:19:50.873236 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.873202 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9"] Oct 30 22:19:50.873840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.873782 4859 cpu_manager.go:414] "containerMap: removing container" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" Oct 30 22:19:50.873840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.873794 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" Oct 30 22:19:50.873840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.873825 4859 cpu_manager.go:414] "containerMap: removing container" podUID="df95d869-36df-46ea-861c-7a01b29cf84f" containerName="maas-api" Oct 30 22:19:50.873840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.873830 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df95d869-36df-46ea-861c-7a01b29cf84f" containerName="maas-api" Oct 30 22:19:50.874113 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.873851 4859 cpu_manager.go:414] "containerMap: removing container" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" Oct 30 22:19:50.874113 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.873857 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" Oct 30 22:19:50.874113 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.873934 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="847c5aab-e0f2-4d3f-8484-77549e797e75" containerName="istio-proxy" Oct 30 22:19:50.874113 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.873956 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef6c3cb-d4e0-4770-8a85-919938bd155a" containerName="istio-proxy" Oct 30 22:19:50.874113 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.873972 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="df95d869-36df-46ea-861c-7a01b29cf84f" containerName="maas-api" Oct 30 22:19:50.877369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.877345 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:50.939820 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.939781 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9"] Oct 30 22:19:50.957794 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.957751 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-model-cache\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:50.958093 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.958067 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-home\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:50.958245 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.958225 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3aa2858-b750-4626-8f15-dfbc009b647f-tls-certs\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:50.958432 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.958414 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-tmp-dir\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:50.958611 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.958551 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8jm\" (UniqueName: \"kubernetes.io/projected/e3aa2858-b750-4626-8f15-dfbc009b647f-kube-api-access-kb8jm\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:50.960075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.958727 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-dshm\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:50.982000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.981966 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2edb49\"/\"default-dockercfg-c4nkh\"" Oct 30 22:19:50.982433 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.982407 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2edb49\"/\"openshift-service-ca.crt\"" Oct 30 22:19:50.982527 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.982457 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2edb49\"/\"ipp-route-2edb49-kserve-self-signed-certs\"" Oct 30 22:19:50.982568 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:50.982555 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2edb49\"/\"kube-root-ca.crt\"" Oct 30 22:19:51.061334 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.061298 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-tmp-dir\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.061541 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.061356 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8jm\" (UniqueName: \"kubernetes.io/projected/e3aa2858-b750-4626-8f15-dfbc009b647f-kube-api-access-kb8jm\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.061541 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.061383 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-dshm\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.061807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.061781 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-tmp-dir\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.061919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.061897 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-model-cache\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.062016 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.061973 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-home\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.062081 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.062019 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3aa2858-b750-4626-8f15-dfbc009b647f-tls-certs\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.062255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.062229 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-model-cache\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.062518 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.062495 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-home\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.063794 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.063763 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-dshm\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.064516 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.064495 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3aa2858-b750-4626-8f15-dfbc009b647f-tls-certs\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.119059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.119029 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8jm\" (UniqueName: \"kubernetes.io/projected/e3aa2858-b750-4626-8f15-dfbc009b647f-kube-api-access-kb8jm\") pod \"ipp-route-2edb49-kserve-7df94fc89-xknd9\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.194879 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.194783 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:51.631972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:51.631932 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9"] Oct 30 22:19:51.633066 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:19:51.633040 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3aa2858_b750_4626_8f15_dfbc009b647f.slice/crio-da707eebcce31ffda7510e9a5e921d15fcf90766ec403f7b6c73f44a395a6eb5 WatchSource:0}: Error finding container da707eebcce31ffda7510e9a5e921d15fcf90766ec403f7b6c73f44a395a6eb5: Status 404 returned error can't find the container with id da707eebcce31ffda7510e9a5e921d15fcf90766ec403f7b6c73f44a395a6eb5 Oct 30 22:19:52.121015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:52.120961 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" event={"ID":"e3aa2858-b750-4626-8f15-dfbc009b647f","Type":"ContainerStarted","Data":"ce802e95ddb863a867d0fdc5558cd99042098403ae3e4dd299b4491b6a07f98f"} Oct 30 22:19:52.121015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:52.121014 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" event={"ID":"e3aa2858-b750-4626-8f15-dfbc009b647f","Type":"ContainerStarted","Data":"da707eebcce31ffda7510e9a5e921d15fcf90766ec403f7b6c73f44a395a6eb5"} Oct 30 22:19:52.195305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:52.195267 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:19:52.237412 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:52.237361 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" podStartSLOduration=2.237348246 podStartE2EDuration="2.237348246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:52.235427043 +0000 UTC m=+2249.608053788" watchObservedRunningTime="2026-10-30 22:19:52.237348246 +0000 UTC m=+2249.609974992" Oct 30 22:19:59.201363 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.201327 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp"] Oct 30 22:19:59.205495 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.205475 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.304645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.299196 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78bb9890-62cb-4557-9871-469bffa798a9-istio-podinfo\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.304645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.299250 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-envoy\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.304645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.299282 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-istio-token\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.304645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.299351 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78bb9890-62cb-4557-9871-469bffa798a9-istiod-ca-cert\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.304645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.299409 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-credential-socket\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.304645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.299449 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-socket\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.304645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.299523 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twlbp\" (UniqueName: \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-kube-api-access-twlbp\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.304645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.299565 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-certs\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.304645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.299617 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-data\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.338296 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.338258 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp"] Oct 30 22:19:59.368389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.368364 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-fe2565-openshift-default-dockercfg-lb9gz\"" Oct 30 22:19:59.400369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400335 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78bb9890-62cb-4557-9871-469bffa798a9-istiod-ca-cert\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.400521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400405 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-credential-socket\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.400521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400436 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-socket\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.400521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400495 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twlbp\" (UniqueName: \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-kube-api-access-twlbp\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.400627 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400533 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-certs\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.400627 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400581 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-data\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.400627 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400621 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78bb9890-62cb-4557-9871-469bffa798a9-istio-podinfo\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.400775 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400744 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-envoy\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.400830 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400798 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-istio-token\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.400891 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400841 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-credential-socket\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.400964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.400897 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-socket\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.401064 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.401031 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-certs\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.401177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.401138 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78bb9890-62cb-4557-9871-469bffa798a9-istiod-ca-cert\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.401241 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.401222 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-data\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.402783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.402762 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-envoy\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.403151 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.403126 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78bb9890-62cb-4557-9871-469bffa798a9-istio-podinfo\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.460278 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.460186 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-istio-token\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.462010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.461982 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twlbp\" (UniqueName: \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-kube-api-access-twlbp\") pod \"e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.521109 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.521080 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:19:59.953240 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.953208 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp"] Oct 30 22:19:59.954028 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:19:59.953997 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bb9890_62cb_4557_9871_469bffa798a9.slice/crio-a3d683fce28f34f4b724a44353a990a20a34e4b9cd980da2cb32f05649a4b7ed WatchSource:0}: Error finding container a3d683fce28f34f4b724a44353a990a20a34e4b9cd980da2cb32f05649a4b7ed: Status 404 returned error can't find the container with id a3d683fce28f34f4b724a44353a990a20a34e4b9cd980da2cb32f05649a4b7ed Oct 30 22:19:59.956809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.956772 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:19:59.956878 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.956852 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:19:59.956915 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:19:59.956893 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:20:00.175517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:00.175484 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" event={"ID":"78bb9890-62cb-4557-9871-469bffa798a9","Type":"ContainerStarted","Data":"eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5"} Oct 30 22:20:00.175696 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:00.175541 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" event={"ID":"78bb9890-62cb-4557-9871-469bffa798a9","Type":"ContainerStarted","Data":"a3d683fce28f34f4b724a44353a990a20a34e4b9cd980da2cb32f05649a4b7ed"} Oct 30 22:20:00.297451 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:00.297332 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" podStartSLOduration=1.29731504 podStartE2EDuration="1.29731504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:00.29380527 +0000 UTC m=+2257.666432015" watchObservedRunningTime="2026-10-30 22:20:00.29731504 +0000 UTC m=+2257.669941786" Oct 30 22:20:00.522110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:00.522082 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:20:01.195800 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:01.195765 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:20:01.208307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:01.208277 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:20:01.527388 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:01.527317 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:20:02.188896 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:02.188860 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:20:02.189843 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:02.189819 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:20:02.199681 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:02.199662 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:20:09.878445 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:20:09.878402 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:20:11.707266 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:20:11.707227 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:20:39.877097 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:20:39.877016 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:21:02.042118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.042081 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8"] Oct 30 22:21:02.042732 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.042318 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" podUID="89c5c684-7187-4ec6-b01a-fb81ef01f7ab" containerName="maas-api" containerID="cri-o://8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8" gracePeriod=30 Oct 30 22:21:02.348282 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.348255 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:21:02.397246 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.397212 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-maas-api-tls\") pod \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " Oct 30 22:21:02.397435 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.397266 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-kube-api-access-xpjpx\" (UniqueName: \"kubernetes.io/projected/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-kube-api-access-xpjpx\") pod \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " Oct 30 22:21:02.397478 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.397450 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-openshift-service-ca\") pod \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " Oct 30 22:21:02.397514 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.397485 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-metrics-certs\") pod \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\" (UID: \"89c5c684-7187-4ec6-b01a-fb81ef01f7ab\") " Oct 30 22:21:02.397834 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.397794 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-openshift-service-ca" pod "89c5c684-7187-4ec6-b01a-fb81ef01f7ab" (UID: "89c5c684-7187-4ec6-b01a-fb81ef01f7ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:02.397994 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.397926 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:02.399518 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.399480 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-maas-api-tls" pod "89c5c684-7187-4ec6-b01a-fb81ef01f7ab" (UID: "89c5c684-7187-4ec6-b01a-fb81ef01f7ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:02.399626 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.399546 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-metrics-certs" pod "89c5c684-7187-4ec6-b01a-fb81ef01f7ab" (UID: "89c5c684-7187-4ec6-b01a-fb81ef01f7ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:02.399626 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.399557 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-kube-api-access-xpjpx" pod "89c5c684-7187-4ec6-b01a-fb81ef01f7ab" (UID: "89c5c684-7187-4ec6-b01a-fb81ef01f7ab"). InnerVolumeSpecName "kube-api-access-xpjpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:02.499224 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.499187 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:02.499224 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.499217 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:02.499497 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.499235 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpjpx\" (UniqueName: \"kubernetes.io/projected/89c5c684-7187-4ec6-b01a-fb81ef01f7ab-kube-api-access-xpjpx\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:02.610401 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.610360 4859 generic.go:356] "Generic (PLEG): container finished" podID="89c5c684-7187-4ec6-b01a-fb81ef01f7ab" containerID="8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8" exitCode=0 Oct 30 22:21:02.610401 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.610401 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" Oct 30 22:21:02.610638 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.610439 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" event={"ID":"89c5c684-7187-4ec6-b01a-fb81ef01f7ab","Type":"ContainerDied","Data":"8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8"} Oct 30 22:21:02.610638 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.610497 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8" event={"ID":"89c5c684-7187-4ec6-b01a-fb81ef01f7ab","Type":"ContainerDied","Data":"1d796c46c65ec45edb009bb74d372965988e3d7168ff7d6fe3476f23196dcb24"} Oct 30 22:21:02.610638 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.610521 4859 scope.go:122] "RemoveContainer" containerID="8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8" Oct 30 22:21:02.622046 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.621884 4859 scope.go:122] "RemoveContainer" containerID="8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8" Oct 30 22:21:02.622340 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:21:02.622318 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8\": container with ID starting with 8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8 not found: ID does not exist" containerID="8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8" Oct 30 22:21:02.622429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.622354 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8"} err="failed to get container status \"8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8\": rpc error: code = NotFound desc = could not find container \"8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8\": container with ID starting with 8878b027ce0a6ce8823bb58f25040cc26761297112c86f923158abcdc1a878d8 not found: ID does not exist" Oct 30 22:21:02.775130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.775090 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8"] Oct 30 22:21:02.778821 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:02.778787 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-577ea8-6c99c57579-hgml8"] Oct 30 22:21:04.743144 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:04.743110 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c5c684-7187-4ec6-b01a-fb81ef01f7ab" path="/var/lib/kubelet/pods/89c5c684-7187-4ec6-b01a-fb81ef01f7ab/volumes" Oct 30 22:21:09.878498 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:21:09.878454 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:21:14.128483 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.128131 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4"] Oct 30 22:21:14.157967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.129183 4859 cpu_manager.go:414] "containerMap: removing container" podUID="89c5c684-7187-4ec6-b01a-fb81ef01f7ab" containerName="maas-api" Oct 30 22:21:14.157967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.129227 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c5c684-7187-4ec6-b01a-fb81ef01f7ab" containerName="maas-api" Oct 30 22:21:14.157967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.129410 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c5c684-7187-4ec6-b01a-fb81ef01f7ab" containerName="maas-api" Oct 30 22:21:14.181967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.167364 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.244970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.242061 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-955887-openshift-default-dockercfg-vkbmh\"" Oct 30 22:21:14.249974 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.249882 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4"] Oct 30 22:21:14.256446 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.256408 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-envoy\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.256732 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.256704 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-certs\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.256896 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.256877 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-istio-token\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.257084 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.257061 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828e5250-0bee-44d5-a646-96e08952ccd3-istio-podinfo\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.257223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.257208 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-credential-socket\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.257350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.257331 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-data\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.257465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.257452 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-socket\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.257618 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.257602 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828e5250-0bee-44d5-a646-96e08952ccd3-istiod-ca-cert\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.257723 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.257709 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxbq\" (UniqueName: \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-kube-api-access-8qxbq\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359065 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-envoy\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359130 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-certs\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359153 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-istio-token\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359276 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828e5250-0bee-44d5-a646-96e08952ccd3-istio-podinfo\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359307 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-credential-socket\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359585 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359341 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-data\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359585 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359378 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-socket\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359585 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359454 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828e5250-0bee-44d5-a646-96e08952ccd3-istiod-ca-cert\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359585 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359491 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxbq\" (UniqueName: \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-kube-api-access-8qxbq\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359608 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-certs\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359639 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-credential-socket\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.359818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.359762 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-data\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.360221 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.360033 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-socket\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.362027 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.360422 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828e5250-0bee-44d5-a646-96e08952ccd3-istiod-ca-cert\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.362027 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.361618 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-envoy\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.362985 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.362589 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828e5250-0bee-44d5-a646-96e08952ccd3-istio-podinfo\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.417406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.417331 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-istio-token\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.417406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.417375 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxbq\" (UniqueName: \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-kube-api-access-8qxbq\") pod \"e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:14.584983 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:14.584914 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:15.017340 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:15.017307 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4"] Oct 30 22:21:15.018372 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:21:15.018341 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828e5250_0bee_44d5_a646_96e08952ccd3.slice/crio-aefac87e90d365a938b541cb9997754890e0e15191f13aaa7f3c14374598ad83 WatchSource:0}: Error finding container aefac87e90d365a938b541cb9997754890e0e15191f13aaa7f3c14374598ad83: Status 404 returned error can't find the container with id aefac87e90d365a938b541cb9997754890e0e15191f13aaa7f3c14374598ad83 Oct 30 22:21:15.020883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:15.020842 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:15.021024 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:15.020970 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:15.021093 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:15.021025 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:15.712548 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:15.712513 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" event={"ID":"828e5250-0bee-44d5-a646-96e08952ccd3","Type":"ContainerStarted","Data":"92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb"} Oct 30 22:21:15.712988 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:15.712581 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" event={"ID":"828e5250-0bee-44d5-a646-96e08952ccd3","Type":"ContainerStarted","Data":"aefac87e90d365a938b541cb9997754890e0e15191f13aaa7f3c14374598ad83"} Oct 30 22:21:15.832606 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:15.832535 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" podStartSLOduration=1.8325173559999999 podStartE2EDuration="1.832517356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:15.832383108 +0000 UTC m=+2333.205009858" watchObservedRunningTime="2026-10-30 22:21:15.832517356 +0000 UTC m=+2333.205144104" Oct 30 22:21:16.310777 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:16.310728 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:21:16.585488 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:16.585414 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:16.591700 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:16.591671 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:16.722000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:16.721962 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:16.723099 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:16.723079 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:21:20.896605 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:20.896571 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv"] Oct 30 22:21:20.901608 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:20.901584 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:20.948016 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:20.947985 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv"] Oct 30 22:21:20.992861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:20.992831 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6e6824-4265-4963-ab7e-e150ee7bcf28-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:20.993059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:20.992870 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:20.993279 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:20.993254 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-metrics-certs\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:20.993350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:20.993296 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqhb\" (UniqueName: \"kubernetes.io/projected/4a6e6824-4265-4963-ab7e-e150ee7bcf28-kube-api-access-vxqhb\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:20.993441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:20.993403 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-fe2565-serving-cert\"" Oct 30 22:21:21.094420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:21.094379 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-metrics-certs\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:21.094613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:21.094429 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqhb\" (UniqueName: \"kubernetes.io/projected/4a6e6824-4265-4963-ab7e-e150ee7bcf28-kube-api-access-vxqhb\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:21.094613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:21.094530 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6e6824-4265-4963-ab7e-e150ee7bcf28-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:21.094613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:21.094562 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:21.094789 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:21:21.094698 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-serving-cert: secret "maas-api-e2e-praxis-b-fe2565-serving-cert" not found Oct 30 22:21:21.094789 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:21:21.094778 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-maas-api-tls podName:4a6e6824-4265-4963-ab7e-e150ee7bcf28 nodeName:}" failed. No retries permitted until 2026-10-30 22:21:21.594756191 +0000 UTC m=+2338.967382931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-maas-api-tls") pod "maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" (UID: "4a6e6824-4265-4963-ab7e-e150ee7bcf28") : secret "maas-api-e2e-praxis-b-fe2565-serving-cert" not found Oct 30 22:21:21.095230 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:21.095205 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6e6824-4265-4963-ab7e-e150ee7bcf28-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:21.096776 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:21.096745 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-metrics-certs\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:21.164235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:21.164156 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqhb\" (UniqueName: \"kubernetes.io/projected/4a6e6824-4265-4963-ab7e-e150ee7bcf28-kube-api-access-vxqhb\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:21.600344 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:21.600307 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:21.602766 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:21.602742 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:21.816889 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:21.816852 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:22.196587 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:22.196554 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv"] Oct 30 22:21:22.197489 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:21:22.197456 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6e6824_4265_4963_ab7e_e150ee7bcf28.slice/crio-d093ca8a4d8ed0daf6fc40d78dbe9a08da7e215181b01fc39f4981117ef1b5f8 WatchSource:0}: Error finding container d093ca8a4d8ed0daf6fc40d78dbe9a08da7e215181b01fc39f4981117ef1b5f8: Status 404 returned error can't find the container with id d093ca8a4d8ed0daf6fc40d78dbe9a08da7e215181b01fc39f4981117ef1b5f8 Oct 30 22:21:22.765648 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:22.765614 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" event={"ID":"4a6e6824-4265-4963-ab7e-e150ee7bcf28","Type":"ContainerStarted","Data":"d093ca8a4d8ed0daf6fc40d78dbe9a08da7e215181b01fc39f4981117ef1b5f8"} Oct 30 22:21:23.775688 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:23.775647 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" event={"ID":"4a6e6824-4265-4963-ab7e-e150ee7bcf28","Type":"ContainerStarted","Data":"5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e"} Oct 30 22:21:23.776134 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:23.775887 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:23.893646 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:23.893545 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" podStartSLOduration=2.981765511 podStartE2EDuration="3.893532941s" totalImagesPullingTime="911.76743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.891184664 +0000 UTC m=+2341.263811411" watchObservedRunningTime="2026-10-30 22:21:23.893532941 +0000 UTC m=+2341.266159687" Oct 30 22:21:29.788156 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:29.788124 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:21:39.903327 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:21:39.903283 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:21:46.959364 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:46.959328 4859 generic.go:356] "Generic (PLEG): container finished" podID="78bb9890-62cb-4557-9871-469bffa798a9" containerID="eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5" exitCode=0 Oct 30 22:21:46.959800 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:46.959404 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" event={"ID":"78bb9890-62cb-4557-9871-469bffa798a9","Type":"ContainerDied","Data":"eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5"} Oct 30 22:21:46.959843 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:46.959823 4859 scope.go:122] "RemoveContainer" containerID="eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5" Oct 30 22:21:46.961614 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:46.961569 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:46.961718 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:46.961670 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:46.961783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:46.961717 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:47.190969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:47.190281 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:21:47.985969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:47.985358 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" event={"ID":"78bb9890-62cb-4557-9871-469bffa798a9","Type":"ContainerStarted","Data":"ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5"} Oct 30 22:21:48.522252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:48.522213 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:21:48.523563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:48.523542 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:21:48.523628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:48.523599 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:21:49.521691 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:49.521656 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:21:49.522135 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:49.521721 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:21:50.522330 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:50.522294 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:21:50.522810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:50.522352 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:21:51.522559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:51.522521 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:21:51.523007 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:51.522575 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:21:52.521896 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:52.521858 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:21:52.522089 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:52.521936 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:21:53.528068 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:53.528033 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:21:53.528466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:53.528419 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:21:53.529142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:21:53.529121 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:22:08.100154 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:08.100069 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:22:09.878231 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:22:09.878192 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:22:23.200313 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:23.200279 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:22:23.213382 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:23.213352 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:22:23.218491 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:23.218455 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:22:23.228447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:23.228424 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:22:25.056961 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.056919 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x"] Oct 30 22:22:25.057402 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.057124 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" podUID="9bddb921-606a-42e6-a25b-40f2ee858f94" containerName="maas-api" containerID="cri-o://ada3e3ad244b113c8547224fa3b56abd2fd8bcfca2f4d1c63e8be2f4f27f8657" gracePeriod=30 Oct 30 22:22:25.329585 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.329499 4859 generic.go:356] "Generic (PLEG): container finished" podID="9bddb921-606a-42e6-a25b-40f2ee858f94" containerID="ada3e3ad244b113c8547224fa3b56abd2fd8bcfca2f4d1c63e8be2f4f27f8657" exitCode=0 Oct 30 22:22:25.329729 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.329581 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" event={"ID":"9bddb921-606a-42e6-a25b-40f2ee858f94","Type":"ContainerDied","Data":"ada3e3ad244b113c8547224fa3b56abd2fd8bcfca2f4d1c63e8be2f4f27f8657"} Oct 30 22:22:25.368586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.368561 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:22:25.442438 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.442403 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-metrics-certs\") pod \"9bddb921-606a-42e6-a25b-40f2ee858f94\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " Oct 30 22:22:25.442894 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.442857 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bddb921-606a-42e6-a25b-40f2ee858f94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bddb921-606a-42e6-a25b-40f2ee858f94-openshift-service-ca\") pod \"9bddb921-606a-42e6-a25b-40f2ee858f94\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " Oct 30 22:22:25.443271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.443250 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-maas-api-tls\") pod \"9bddb921-606a-42e6-a25b-40f2ee858f94\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " Oct 30 22:22:25.443424 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.443409 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bddb921-606a-42e6-a25b-40f2ee858f94-kube-api-access-xwbwz\" (UniqueName: \"kubernetes.io/projected/9bddb921-606a-42e6-a25b-40f2ee858f94-kube-api-access-xwbwz\") pod \"9bddb921-606a-42e6-a25b-40f2ee858f94\" (UID: \"9bddb921-606a-42e6-a25b-40f2ee858f94\") " Oct 30 22:22:25.473973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.448185 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bddb921-606a-42e6-a25b-40f2ee858f94-openshift-service-ca" pod "9bddb921-606a-42e6-a25b-40f2ee858f94" (UID: "9bddb921-606a-42e6-a25b-40f2ee858f94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:25.474725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.474585 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-maas-api-tls" pod "9bddb921-606a-42e6-a25b-40f2ee858f94" (UID: "9bddb921-606a-42e6-a25b-40f2ee858f94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:22:25.474725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.474609 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bddb921-606a-42e6-a25b-40f2ee858f94-kube-api-access-xwbwz" pod "9bddb921-606a-42e6-a25b-40f2ee858f94" (UID: "9bddb921-606a-42e6-a25b-40f2ee858f94"). InnerVolumeSpecName "kube-api-access-xwbwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:25.474725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.474691 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-metrics-certs" pod "9bddb921-606a-42e6-a25b-40f2ee858f94" (UID: "9bddb921-606a-42e6-a25b-40f2ee858f94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:22:25.545041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.545010 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:25.545041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.545037 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwbwz\" (UniqueName: \"kubernetes.io/projected/9bddb921-606a-42e6-a25b-40f2ee858f94-kube-api-access-xwbwz\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:25.545041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.545047 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bddb921-606a-42e6-a25b-40f2ee858f94-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:25.545275 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:25.545056 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bddb921-606a-42e6-a25b-40f2ee858f94-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:26.365862 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:26.365822 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" event={"ID":"9bddb921-606a-42e6-a25b-40f2ee858f94","Type":"ContainerDied","Data":"07f617a25f02aa1c4c4d50c88e7f4b92e501bfbeeaddcd275b165f41b61d144e"} Oct 30 22:22:26.368960 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:26.368919 4859 scope.go:122] "RemoveContainer" containerID="ada3e3ad244b113c8547224fa3b56abd2fd8bcfca2f4d1c63e8be2f4f27f8657" Oct 30 22:22:26.369309 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:26.369291 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x" Oct 30 22:22:26.541700 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:26.541659 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x"] Oct 30 22:22:26.591059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:26.591024 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-69a1d2-54b54f868d-69d9x"] Oct 30 22:22:26.742302 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:26.742223 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bddb921-606a-42e6-a25b-40f2ee858f94" path="/var/lib/kubelet/pods/9bddb921-606a-42e6-a25b-40f2ee858f94/volumes" Oct 30 22:22:34.681587 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:34.681552 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh"] Oct 30 22:22:34.682105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:34.681843 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" containerID="cri-o://321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049" gracePeriod=30 Oct 30 22:22:39.883726 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:22:39.883689 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:22:39.967445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:39.967421 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:22:40.029219 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029187 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-certs\") pod \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " Oct 30 22:22:40.029396 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029277 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-credential-socket\") pod \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " Oct 30 22:22:40.029396 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029361 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-envoy\") pod \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " Oct 30 22:22:40.029519 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029396 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-data\") pod \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " Oct 30 22:22:40.029519 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029444 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-certs" pod "3c25e8c3-71db-44fe-9c39-85ced3c3aa79" (UID: "3c25e8c3-71db-44fe-9c39-85ced3c3aa79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:40.029519 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029478 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istiod-ca-cert\") pod \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " Oct 30 22:22:40.029673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029531 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-token\" (UniqueName: \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-token\") pod \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " Oct 30 22:22:40.029673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029557 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-credential-socket" pod "3c25e8c3-71db-44fe-9c39-85ced3c3aa79" (UID: "3c25e8c3-71db-44fe-9c39-85ced3c3aa79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:40.029673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029625 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-data" pod "3c25e8c3-71db-44fe-9c39-85ced3c3aa79" (UID: "3c25e8c3-71db-44fe-9c39-85ced3c3aa79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:40.029673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029663 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-socket\") pod \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " Oct 30 22:22:40.029883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029718 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-kube-api-access-s5b2h\" (UniqueName: \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-kube-api-access-s5b2h\") pod \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " Oct 30 22:22:40.029883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029743 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-podinfo\") pod \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\" (UID: \"3c25e8c3-71db-44fe-9c39-85ced3c3aa79\") " Oct 30 22:22:40.030041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029932 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-socket" pod "3c25e8c3-71db-44fe-9c39-85ced3c3aa79" (UID: "3c25e8c3-71db-44fe-9c39-85ced3c3aa79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:40.030041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.029982 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istiod-ca-cert" pod "3c25e8c3-71db-44fe-9c39-85ced3c3aa79" (UID: "3c25e8c3-71db-44fe-9c39-85ced3c3aa79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:40.030356 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.030334 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:40.030450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.030361 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:40.030450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.030377 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:40.030450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.030391 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:40.030450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.030405 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:40.031617 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.031588 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-envoy" pod "3c25e8c3-71db-44fe-9c39-85ced3c3aa79" (UID: "3c25e8c3-71db-44fe-9c39-85ced3c3aa79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:40.031762 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.031743 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-token" pod "3c25e8c3-71db-44fe-9c39-85ced3c3aa79" (UID: "3c25e8c3-71db-44fe-9c39-85ced3c3aa79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:40.031849 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.031833 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-podinfo" pod "3c25e8c3-71db-44fe-9c39-85ced3c3aa79" (UID: "3c25e8c3-71db-44fe-9c39-85ced3c3aa79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:22:40.031925 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.031908 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-kube-api-access-s5b2h" pod "3c25e8c3-71db-44fe-9c39-85ced3c3aa79" (UID: "3c25e8c3-71db-44fe-9c39-85ced3c3aa79"). InnerVolumeSpecName "kube-api-access-s5b2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:40.131663 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.131620 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:40.131663 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.131651 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:40.131663 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.131663 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5b2h\" (UniqueName: \"kubernetes.io/projected/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-kube-api-access-s5b2h\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:40.131663 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.131673 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c25e8c3-71db-44fe-9c39-85ced3c3aa79-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:40.475651 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.475620 4859 generic.go:356] "Generic (PLEG): container finished" podID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerID="321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049" exitCode=0 Oct 30 22:22:40.475861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.475683 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" Oct 30 22:22:40.475861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.475707 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" event={"ID":"3c25e8c3-71db-44fe-9c39-85ced3c3aa79","Type":"ContainerDied","Data":"321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049"} Oct 30 22:22:40.475861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.475775 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh" event={"ID":"3c25e8c3-71db-44fe-9c39-85ced3c3aa79","Type":"ContainerDied","Data":"ace12f6b346f34ced528d45d6899a63f0623962385f1d246a2ec3ed195d5f09e"} Oct 30 22:22:40.475861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.475808 4859 scope.go:122] "RemoveContainer" containerID="321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049" Oct 30 22:22:40.490376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.490350 4859 scope.go:122] "RemoveContainer" containerID="c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a" Oct 30 22:22:40.501527 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.501508 4859 scope.go:122] "RemoveContainer" containerID="321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049" Oct 30 22:22:40.501814 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:22:40.501789 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049\": container with ID starting with 321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049 not found: ID does not exist" containerID="321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049" Oct 30 22:22:40.501882 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.501827 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049"} err="failed to get container status \"321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049\": rpc error: code = NotFound desc = could not find container \"321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049\": container with ID starting with 321485bfaf421510a3675677e0149a77095a11426c7d5c6ec0f85b854dcd0049 not found: ID does not exist" Oct 30 22:22:40.501882 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.501854 4859 scope.go:122] "RemoveContainer" containerID="c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a" Oct 30 22:22:40.502146 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:22:40.502128 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a\": container with ID starting with c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a not found: ID does not exist" containerID="c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a" Oct 30 22:22:40.502214 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.502152 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a"} err="failed to get container status \"c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a\": rpc error: code = NotFound desc = could not find container \"c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a\": container with ID starting with c0df37e64f0703bf919b6e9d5de4c002988ddc8dcc721f5a0128c511d56adb3a not found: ID does not exist" Oct 30 22:22:40.639741 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.639708 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh"] Oct 30 22:22:40.688506 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.688475 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a1d2-openshift-default-987788949-sshrh"] Oct 30 22:22:40.741600 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:40.741529 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" path="/var/lib/kubelet/pods/3c25e8c3-71db-44fe-9c39-85ced3c3aa79/volumes" Oct 30 22:22:42.709314 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.709279 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6"] Oct 30 22:22:42.709870 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.709853 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" Oct 30 22:22:42.709913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.709873 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" Oct 30 22:22:42.709913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.709890 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" Oct 30 22:22:42.709913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.709896 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" Oct 30 22:22:42.709913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.709906 4859 cpu_manager.go:414] "containerMap: removing container" podUID="9bddb921-606a-42e6-a25b-40f2ee858f94" containerName="maas-api" Oct 30 22:22:42.709913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.709911 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bddb921-606a-42e6-a25b-40f2ee858f94" containerName="maas-api" Oct 30 22:22:42.710102 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.710034 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" Oct 30 22:22:42.710102 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.710042 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bddb921-606a-42e6-a25b-40f2ee858f94" containerName="maas-api" Oct 30 22:22:42.710102 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.710059 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c25e8c3-71db-44fe-9c39-85ced3c3aa79" containerName="istio-proxy" Oct 30 22:22:42.714472 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.714451 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.762725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.762682 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0b05b-f073-4643-ab5d-6771be17eeb1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.762933 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.762916 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.763034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.763009 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-metrics-certs\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.763210 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.763187 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vzw\" (UniqueName: \"kubernetes.io/projected/6ac0b05b-f073-4643-ab5d-6771be17eeb1-kube-api-access-96vzw\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.766448 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.766428 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-955887-serving-cert\"" Oct 30 22:22:42.818992 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.818956 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6"] Oct 30 22:22:42.864254 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.864223 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.864457 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.864274 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-metrics-certs\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.864457 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.864357 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vzw\" (UniqueName: \"kubernetes.io/projected/6ac0b05b-f073-4643-ab5d-6771be17eeb1-kube-api-access-96vzw\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.864457 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:22:42.864389 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-serving-cert: secret "maas-api-e2e-ipp-b-955887-serving-cert" not found Oct 30 22:22:42.864457 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.864414 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0b05b-f073-4643-ab5d-6771be17eeb1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.864680 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:22:42.864471 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-maas-api-tls podName:6ac0b05b-f073-4643-ab5d-6771be17eeb1 nodeName:}" failed. No retries permitted until 2026-10-30 22:22:43.364450447 +0000 UTC m=+2420.737077175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-maas-api-tls") pod "maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" (UID: "6ac0b05b-f073-4643-ab5d-6771be17eeb1") : secret "maas-api-e2e-ipp-b-955887-serving-cert" not found Oct 30 22:22:42.865256 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.865219 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0b05b-f073-4643-ab5d-6771be17eeb1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.867059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.867032 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-metrics-certs\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:42.925241 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:42.925181 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vzw\" (UniqueName: \"kubernetes.io/projected/6ac0b05b-f073-4643-ab5d-6771be17eeb1-kube-api-access-96vzw\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:43.393399 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:43.393360 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:43.395853 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:43.395816 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:43.628170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:43.628134 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:44.015640 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:44.015607 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6"] Oct 30 22:22:44.017268 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:22:44.017233 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac0b05b_f073_4643_ab5d_6771be17eeb1.slice/crio-5392db00a18ddcbac8c35bd68e70c57e41c4cef0f7c6b36f6c5c18e2e67e5723 WatchSource:0}: Error finding container 5392db00a18ddcbac8c35bd68e70c57e41c4cef0f7c6b36f6c5c18e2e67e5723: Status 404 returned error can't find the container with id 5392db00a18ddcbac8c35bd68e70c57e41c4cef0f7c6b36f6c5c18e2e67e5723 Oct 30 22:22:44.519086 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:44.519047 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" event={"ID":"6ac0b05b-f073-4643-ab5d-6771be17eeb1","Type":"ContainerStarted","Data":"5392db00a18ddcbac8c35bd68e70c57e41c4cef0f7c6b36f6c5c18e2e67e5723"} Oct 30 22:22:45.528393 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:45.528352 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" event={"ID":"6ac0b05b-f073-4643-ab5d-6771be17eeb1","Type":"ContainerStarted","Data":"2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63"} Oct 30 22:22:45.528857 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:45.528561 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:22:45.645474 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:45.645413 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" podStartSLOduration=2.8206603230000002 podStartE2EDuration="3.645396136s" totalImagesPullingTime="824.735813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:45.644202504 +0000 UTC m=+2423.016829251" watchObservedRunningTime="2026-10-30 22:22:45.645396136 +0000 UTC m=+2423.018022883" Oct 30 22:22:51.542299 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:22:51.542265 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:23:08.418928 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:08.418894 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:23:09.289025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.288992 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v"] Oct 30 22:23:09.293288 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.293269 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.347581 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.347547 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-cce50a-openshift-default-dockercfg-49cjl\"" Oct 30 22:23:09.410866 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.410829 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v"] Oct 30 22:23:09.417038 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.416759 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d209f91-4095-47a2-866d-d58f89e198a2-istiod-ca-cert\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.417038 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.416901 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrd8l\" (UniqueName: \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-kube-api-access-hrd8l\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.417038 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.416936 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-socket\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.417346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.417094 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-istio-token\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.417346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.417126 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-envoy\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.417346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.417193 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-certs\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.417346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.417229 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d209f91-4095-47a2-866d-d58f89e198a2-istio-podinfo\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.417346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.417259 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-credential-socket\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.417346 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.417316 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-data\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.522045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.521828 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-certs\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.522045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.521891 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d209f91-4095-47a2-866d-d58f89e198a2-istio-podinfo\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.522045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.521922 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-credential-socket\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.522045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.521995 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-data\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.522704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.522069 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d209f91-4095-47a2-866d-d58f89e198a2-istiod-ca-cert\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.522704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.522184 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrd8l\" (UniqueName: \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-kube-api-access-hrd8l\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.522704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.522213 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-socket\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.522704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.522322 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-istio-token\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.522704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.522350 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-envoy\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.529966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.523161 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-data\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.529966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.523432 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-certs\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.529966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.525226 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-credential-socket\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.529966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.525250 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d209f91-4095-47a2-866d-d58f89e198a2-istiod-ca-cert\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.529966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.525462 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-socket\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.541012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.535595 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d209f91-4095-47a2-866d-d58f89e198a2-istio-podinfo\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.541012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.535854 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-envoy\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.596966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.596687 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-istio-token\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.600964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.598280 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrd8l\" (UniqueName: \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-kube-api-access-hrd8l\") pod \"e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.612981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:09.607196 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:09.903544 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:23:09.903157 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:23:10.091220 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:10.091184 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v"] Oct 30 22:23:10.130123 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:23:10.130093 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d209f91_4095_47a2_866d_d58f89e198a2.slice/crio-b561d982682c9298d2516a9449e5185ad9ec45cc4adc71a7d49f1510dc26cd79 WatchSource:0}: Error finding container b561d982682c9298d2516a9449e5185ad9ec45cc4adc71a7d49f1510dc26cd79: Status 404 returned error can't find the container with id b561d982682c9298d2516a9449e5185ad9ec45cc4adc71a7d49f1510dc26cd79 Oct 30 22:23:10.132679 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:10.132644 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:23:10.132792 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:10.132716 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:23:10.132792 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:10.132747 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:23:10.757149 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:10.757110 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" event={"ID":"5d209f91-4095-47a2-866d-d58f89e198a2","Type":"ContainerStarted","Data":"bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3"} Oct 30 22:23:10.757532 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:10.757168 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" event={"ID":"5d209f91-4095-47a2-866d-d58f89e198a2","Type":"ContainerStarted","Data":"b561d982682c9298d2516a9449e5185ad9ec45cc4adc71a7d49f1510dc26cd79"} Oct 30 22:23:10.890066 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:10.890009 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" podStartSLOduration=1.88999601 podStartE2EDuration="1.88999601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:10.886521823 +0000 UTC m=+2448.259148570" watchObservedRunningTime="2026-10-30 22:23:10.88999601 +0000 UTC m=+2448.262622755" Oct 30 22:23:11.608256 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:11.608220 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:11.614301 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:11.614270 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:11.769099 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:11.769061 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:11.786727 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:11.786701 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:23:17.763124 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:17.763076 4859 patch_prober.go:29] interesting pod/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:17.763589 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:17.763154 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:18.844647 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:18.844618 4859 generic.go:356] "Generic (PLEG): container finished" podID="828e5250-0bee-44d5-a646-96e08952ccd3" containerID="92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb" exitCode=0 Oct 30 22:23:18.845102 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:18.844690 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" event={"ID":"828e5250-0bee-44d5-a646-96e08952ccd3","Type":"ContainerDied","Data":"92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb"} Oct 30 22:23:18.845102 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:18.845072 4859 scope.go:122] "RemoveContainer" containerID="92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb" Oct 30 22:23:18.846527 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:18.846497 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:23:18.846648 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:18.846585 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:23:18.846648 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:18.846637 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:23:19.855281 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:19.855246 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" event={"ID":"828e5250-0bee-44d5-a646-96e08952ccd3","Type":"ContainerStarted","Data":"4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4"} Oct 30 22:23:20.586169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:20.586140 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:23:20.586466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:20.586342 4859 patch_prober.go:29] interesting pod/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:23:20.586578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:20.586491 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:23:21.586329 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:21.586291 4859 patch_prober.go:29] interesting pod/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:23:21.586852 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:21.586349 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:23:22.586374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:22.586333 4859 patch_prober.go:29] interesting pod/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:23:22.586747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:22.586410 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:23:23.585352 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:23.585313 4859 patch_prober.go:29] interesting pod/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:23:23.585640 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:23.585612 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:23:24.586334 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:24.586297 4859 patch_prober.go:29] interesting pod/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:23:24.586726 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:24.586353 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:23:25.590370 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:25.590329 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:23:25.590766 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:25.590736 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:23:25.591472 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:25.591448 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:23:39.880056 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:23:39.879963 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:23:41.039602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:23:41.039561 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:24:00.340436 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.340397 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6"] Oct 30 22:24:00.341003 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.340596 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" podUID="f454daf2-9772-4eba-81fc-d9acf70939ea" containerName="maas-api" containerID="cri-o://72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39" gracePeriod=30 Oct 30 22:24:00.636657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.636629 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:24:00.697035 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.697002 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-maas-api-tls\") pod \"f454daf2-9772-4eba-81fc-d9acf70939ea\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " Oct 30 22:24:00.697202 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.697107 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f454daf2-9772-4eba-81fc-d9acf70939ea-kube-api-access-82x9x\" (UniqueName: \"kubernetes.io/projected/f454daf2-9772-4eba-81fc-d9acf70939ea-kube-api-access-82x9x\") pod \"f454daf2-9772-4eba-81fc-d9acf70939ea\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " Oct 30 22:24:00.697383 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.697365 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-metrics-certs\") pod \"f454daf2-9772-4eba-81fc-d9acf70939ea\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " Oct 30 22:24:00.697444 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.697422 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f454daf2-9772-4eba-81fc-d9acf70939ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454daf2-9772-4eba-81fc-d9acf70939ea-openshift-service-ca\") pod \"f454daf2-9772-4eba-81fc-d9acf70939ea\" (UID: \"f454daf2-9772-4eba-81fc-d9acf70939ea\") " Oct 30 22:24:00.697828 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.697797 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f454daf2-9772-4eba-81fc-d9acf70939ea-openshift-service-ca" pod "f454daf2-9772-4eba-81fc-d9acf70939ea" (UID: "f454daf2-9772-4eba-81fc-d9acf70939ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:00.698004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.697979 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454daf2-9772-4eba-81fc-d9acf70939ea-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:00.699721 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.699549 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-metrics-certs" pod "f454daf2-9772-4eba-81fc-d9acf70939ea" (UID: "f454daf2-9772-4eba-81fc-d9acf70939ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:00.699721 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.699644 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-maas-api-tls" pod "f454daf2-9772-4eba-81fc-d9acf70939ea" (UID: "f454daf2-9772-4eba-81fc-d9acf70939ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:00.699721 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.699689 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f454daf2-9772-4eba-81fc-d9acf70939ea-kube-api-access-82x9x" pod "f454daf2-9772-4eba-81fc-d9acf70939ea" (UID: "f454daf2-9772-4eba-81fc-d9acf70939ea"). InnerVolumeSpecName "kube-api-access-82x9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:00.798835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.798805 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:00.798835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.798836 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f454daf2-9772-4eba-81fc-d9acf70939ea-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:00.799047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:00.798849 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82x9x\" (UniqueName: \"kubernetes.io/projected/f454daf2-9772-4eba-81fc-d9acf70939ea-kube-api-access-82x9x\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:01.224676 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:01.224644 4859 generic.go:356] "Generic (PLEG): container finished" podID="f454daf2-9772-4eba-81fc-d9acf70939ea" containerID="72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39" exitCode=0 Oct 30 22:24:01.224844 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:01.224701 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" Oct 30 22:24:01.224844 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:01.224710 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" event={"ID":"f454daf2-9772-4eba-81fc-d9acf70939ea","Type":"ContainerDied","Data":"72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39"} Oct 30 22:24:01.224844 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:01.224778 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6" event={"ID":"f454daf2-9772-4eba-81fc-d9acf70939ea","Type":"ContainerDied","Data":"f27db2a325a6013cb8da239d8296df53e73adc23b75a2b8451eca7700317cc54"} Oct 30 22:24:01.224844 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:01.224800 4859 scope.go:122] "RemoveContainer" containerID="72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39" Oct 30 22:24:01.235046 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:01.235028 4859 scope.go:122] "RemoveContainer" containerID="72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39" Oct 30 22:24:01.235327 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:24:01.235305 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39\": container with ID starting with 72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39 not found: ID does not exist" containerID="72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39" Oct 30 22:24:01.235381 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:01.235338 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39"} err="failed to get container status \"72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39\": rpc error: code = NotFound desc = could not find container \"72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39\": container with ID starting with 72e46acafe038e1fb8e7af48b6a6cf850b4d98d5466e3700ff8a8d3a82281d39 not found: ID does not exist" Oct 30 22:24:01.460043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:01.460010 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6"] Oct 30 22:24:01.467892 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:01.467859 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2edb49-64dbbd67fc-pp4n6"] Oct 30 22:24:02.742326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:02.742290 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f454daf2-9772-4eba-81fc-d9acf70939ea" path="/var/lib/kubelet/pods/f454daf2-9772-4eba-81fc-d9acf70939ea/volumes" Oct 30 22:24:05.959145 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:05.959113 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m"] Oct 30 22:24:05.959560 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:05.959320 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" containerID="cri-o://4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f" gracePeriod=30 Oct 30 22:24:08.072748 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:08.072714 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9"] Oct 30 22:24:09.912181 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:24:09.912136 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:24:10.607019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:10.606987 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw"] Oct 30 22:24:10.607217 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:10.607195 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerName="istio-proxy" containerID="cri-o://1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527" gracePeriod=30 Oct 30 22:24:11.217364 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.217341 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:24:11.298022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.297902 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/571531de-dec4-474a-b197-8d84823a8198-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571531de-dec4-474a-b197-8d84823a8198-istio-podinfo\") pod \"571531de-dec4-474a-b197-8d84823a8198\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " Oct 30 22:24:11.298238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.298220 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-istio-token\" (UniqueName: \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-istio-token\") pod \"571531de-dec4-474a-b197-8d84823a8198\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " Oct 30 22:24:11.298309 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.298265 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/571531de-dec4-474a-b197-8d84823a8198-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571531de-dec4-474a-b197-8d84823a8198-istiod-ca-cert\") pod \"571531de-dec4-474a-b197-8d84823a8198\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " Oct 30 22:24:11.298370 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.298318 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-socket\") pod \"571531de-dec4-474a-b197-8d84823a8198\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " Oct 30 22:24:11.298618 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.298593 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-socket" pod "571531de-dec4-474a-b197-8d84823a8198" (UID: "571531de-dec4-474a-b197-8d84823a8198"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:11.298711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.298654 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-certs\") pod \"571531de-dec4-474a-b197-8d84823a8198\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " Oct 30 22:24:11.298711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.298686 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571531de-dec4-474a-b197-8d84823a8198-istiod-ca-cert" pod "571531de-dec4-474a-b197-8d84823a8198" (UID: "571531de-dec4-474a-b197-8d84823a8198"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:11.298817 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.298694 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-envoy\") pod \"571531de-dec4-474a-b197-8d84823a8198\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " Oct 30 22:24:11.298817 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.298756 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-data\") pod \"571531de-dec4-474a-b197-8d84823a8198\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " Oct 30 22:24:11.298817 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.298811 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-kube-api-access-mwd46\" (UniqueName: \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-kube-api-access-mwd46\") pod \"571531de-dec4-474a-b197-8d84823a8198\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " Oct 30 22:24:11.298986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.298837 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-credential-socket\") pod \"571531de-dec4-474a-b197-8d84823a8198\" (UID: \"571531de-dec4-474a-b197-8d84823a8198\") " Oct 30 22:24:11.299842 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.299533 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:11.299842 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.299560 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571531de-dec4-474a-b197-8d84823a8198-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:11.299842 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.299538 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-data" pod "571531de-dec4-474a-b197-8d84823a8198" (UID: "571531de-dec4-474a-b197-8d84823a8198"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:11.299842 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.299766 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-credential-socket" pod "571531de-dec4-474a-b197-8d84823a8198" (UID: "571531de-dec4-474a-b197-8d84823a8198"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:11.299842 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.299788 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-certs" pod "571531de-dec4-474a-b197-8d84823a8198" (UID: "571531de-dec4-474a-b197-8d84823a8198"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:11.301138 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.301115 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-envoy" pod "571531de-dec4-474a-b197-8d84823a8198" (UID: "571531de-dec4-474a-b197-8d84823a8198"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:11.301264 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.301170 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-istio-token" pod "571531de-dec4-474a-b197-8d84823a8198" (UID: "571531de-dec4-474a-b197-8d84823a8198"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:11.301385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.301346 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/571531de-dec4-474a-b197-8d84823a8198-istio-podinfo" pod "571531de-dec4-474a-b197-8d84823a8198" (UID: "571531de-dec4-474a-b197-8d84823a8198"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:11.301750 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.301727 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-kube-api-access-mwd46" pod "571531de-dec4-474a-b197-8d84823a8198" (UID: "571531de-dec4-474a-b197-8d84823a8198"). InnerVolumeSpecName "kube-api-access-mwd46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:11.306693 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.306668 4859 generic.go:356] "Generic (PLEG): container finished" podID="571531de-dec4-474a-b197-8d84823a8198" containerID="4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f" exitCode=0 Oct 30 22:24:11.306793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.306724 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" Oct 30 22:24:11.306793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.306723 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" event={"ID":"571531de-dec4-474a-b197-8d84823a8198","Type":"ContainerDied","Data":"4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f"} Oct 30 22:24:11.306793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.306790 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m" event={"ID":"571531de-dec4-474a-b197-8d84823a8198","Type":"ContainerDied","Data":"ef7ec32908470f315324e96daa352028a6fbd2db7a5460a7bd81cb1dcc30b303"} Oct 30 22:24:11.307052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.306822 4859 scope.go:122] "RemoveContainer" containerID="4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f" Oct 30 22:24:11.329117 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.329097 4859 scope.go:122] "RemoveContainer" containerID="4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f" Oct 30 22:24:11.329382 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:24:11.329365 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f\": container with ID starting with 4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f not found: ID does not exist" containerID="4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f" Oct 30 22:24:11.329442 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.329388 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f"} err="failed to get container status \"4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f\": rpc error: code = NotFound desc = could not find container \"4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f\": container with ID starting with 4e6c4197adca4e4bc029e16f1188c879511aeaf9e76b3d255b6ba6508b057a3f not found: ID does not exist" Oct 30 22:24:11.401276 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.401240 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571531de-dec4-474a-b197-8d84823a8198-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:11.401276 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.401270 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:11.401465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.401297 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:11.401465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.401311 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:11.401465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.401323 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:11.401465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.401335 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwd46\" (UniqueName: \"kubernetes.io/projected/571531de-dec4-474a-b197-8d84823a8198-kube-api-access-mwd46\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:11.401465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.401347 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571531de-dec4-474a-b197-8d84823a8198-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:11.453011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.452980 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m"] Oct 30 22:24:11.456052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:11.456031 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msm7m"] Oct 30 22:24:12.743525 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:12.743492 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571531de-dec4-474a-b197-8d84823a8198" path="/var/lib/kubelet/pods/571531de-dec4-474a-b197-8d84823a8198/volumes" Oct 30 22:24:15.867797 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.867769 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 22:24:15.954202 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954145 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-certs\") pod \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " Oct 30 22:24:15.954202 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954177 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istiod-ca-cert\") pod \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " Oct 30 22:24:15.954411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954205 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-kube-api-access-tt4l6\" (UniqueName: \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-kube-api-access-tt4l6\") pod \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " Oct 30 22:24:15.954411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954226 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-token\" (UniqueName: \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-token\") pod \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " Oct 30 22:24:15.954411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954332 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-podinfo\") pod \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " Oct 30 22:24:15.954411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954368 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-envoy\") pod \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " Oct 30 22:24:15.954411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954395 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-certs" pod "a77f54c3-1e05-4549-bdd9-d2a5a768d84f" (UID: "a77f54c3-1e05-4549-bdd9-d2a5a768d84f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:15.954714 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954508 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-data\") pod \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " Oct 30 22:24:15.954714 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954563 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-credential-socket\") pod \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " Oct 30 22:24:15.954714 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954606 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-socket\") pod \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\" (UID: \"a77f54c3-1e05-4549-bdd9-d2a5a768d84f\") " Oct 30 22:24:15.954714 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954608 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istiod-ca-cert" pod "a77f54c3-1e05-4549-bdd9-d2a5a768d84f" (UID: "a77f54c3-1e05-4549-bdd9-d2a5a768d84f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:15.954905 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954850 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-data" pod "a77f54c3-1e05-4549-bdd9-d2a5a768d84f" (UID: "a77f54c3-1e05-4549-bdd9-d2a5a768d84f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:15.955002 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.954906 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-credential-socket" pod "a77f54c3-1e05-4549-bdd9-d2a5a768d84f" (UID: "a77f54c3-1e05-4549-bdd9-d2a5a768d84f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:15.955063 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.955032 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-socket" pod "a77f54c3-1e05-4549-bdd9-d2a5a768d84f" (UID: "a77f54c3-1e05-4549-bdd9-d2a5a768d84f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:15.955164 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.955148 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.955238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.955167 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.955238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.955181 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.955238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.955202 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.955238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.955214 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.956561 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.956537 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-podinfo" pod "a77f54c3-1e05-4549-bdd9-d2a5a768d84f" (UID: "a77f54c3-1e05-4549-bdd9-d2a5a768d84f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:15.956668 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.956651 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-kube-api-access-tt4l6" pod "a77f54c3-1e05-4549-bdd9-d2a5a768d84f" (UID: "a77f54c3-1e05-4549-bdd9-d2a5a768d84f"). InnerVolumeSpecName "kube-api-access-tt4l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:15.956809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.956789 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-envoy" pod "a77f54c3-1e05-4549-bdd9-d2a5a768d84f" (UID: "a77f54c3-1e05-4549-bdd9-d2a5a768d84f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:15.956861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:15.956838 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-token" pod "a77f54c3-1e05-4549-bdd9-d2a5a768d84f" (UID: "a77f54c3-1e05-4549-bdd9-d2a5a768d84f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:16.056807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.056783 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:16.056807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.056807 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:16.056973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.056817 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt4l6\" (UniqueName: \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-kube-api-access-tt4l6\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:16.056973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.056826 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77f54c3-1e05-4549-bdd9-d2a5a768d84f-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:16.344105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.344072 4859 generic.go:356] "Generic (PLEG): container finished" podID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerID="1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527" exitCode=0 Oct 30 22:24:16.344279 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.344130 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" Oct 30 22:24:16.344279 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.344165 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" event={"ID":"a77f54c3-1e05-4549-bdd9-d2a5a768d84f","Type":"ContainerDied","Data":"1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527"} Oct 30 22:24:16.344279 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.344220 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw" event={"ID":"a77f54c3-1e05-4549-bdd9-d2a5a768d84f","Type":"ContainerDied","Data":"9a3ea2e1e59774f7024752342dccb3166fb385185171deb831ec494c9aab012f"} Oct 30 22:24:16.344279 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.344242 4859 scope.go:122] "RemoveContainer" containerID="1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527" Oct 30 22:24:16.354979 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.354957 4859 scope.go:122] "RemoveContainer" containerID="1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527" Oct 30 22:24:16.355300 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:24:16.355274 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527\": container with ID starting with 1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527 not found: ID does not exist" containerID="1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527" Oct 30 22:24:16.355391 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.355309 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527"} err="failed to get container status \"1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527\": rpc error: code = NotFound desc = could not find container \"1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527\": container with ID starting with 1dc6be2fa3b225de011e7835a25d81f801f5368b0b462904b0a571aad741d527 not found: ID does not exist" Oct 30 22:24:16.511955 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.511920 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw"] Oct 30 22:24:16.517371 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.517346 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-vf6jw"] Oct 30 22:24:16.741767 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:16.741731 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" path="/var/lib/kubelet/pods/a77f54c3-1e05-4549-bdd9-d2a5a768d84f/volumes" Oct 30 22:24:23.093245 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.093173 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" podUID="e3aa2858-b750-4626-8f15-dfbc009b647f" containerName="main" containerID="cri-o://ce802e95ddb863a867d0fdc5558cd99042098403ae3e4dd299b4491b6a07f98f" gracePeriod=45 Oct 30 22:24:23.189580 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.189535 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" podUID="e3aa2858-b750-4626-8f15-dfbc009b647f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": dial tcp 10.132.0.133:8000: connect: connection refused" Oct 30 22:24:23.437462 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.437085 4859 generic.go:356] "Generic (PLEG): container finished" podID="e3aa2858-b750-4626-8f15-dfbc009b647f" containerID="ce802e95ddb863a867d0fdc5558cd99042098403ae3e4dd299b4491b6a07f98f" exitCode=0 Oct 30 22:24:23.437462 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.437300 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" event={"ID":"e3aa2858-b750-4626-8f15-dfbc009b647f","Type":"ContainerDied","Data":"ce802e95ddb863a867d0fdc5558cd99042098403ae3e4dd299b4491b6a07f98f"} Oct 30 22:24:23.480312 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.480287 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:24:23.581046 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581003 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-model-cache\") pod \"e3aa2858-b750-4626-8f15-dfbc009b647f\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " Oct 30 22:24:23.581212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581117 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3aa2858-b750-4626-8f15-dfbc009b647f-kube-api-access-kb8jm\" (UniqueName: \"kubernetes.io/projected/e3aa2858-b750-4626-8f15-dfbc009b647f-kube-api-access-kb8jm\") pod \"e3aa2858-b750-4626-8f15-dfbc009b647f\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " Oct 30 22:24:23.581259 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581216 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-dshm\") pod \"e3aa2858-b750-4626-8f15-dfbc009b647f\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " Oct 30 22:24:23.581299 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581275 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-home\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-home\") pod \"e3aa2858-b750-4626-8f15-dfbc009b647f\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " Oct 30 22:24:23.581299 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581276 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-model-cache" pod "e3aa2858-b750-4626-8f15-dfbc009b647f" (UID: "e3aa2858-b750-4626-8f15-dfbc009b647f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:23.581378 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581313 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3aa2858-b750-4626-8f15-dfbc009b647f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3aa2858-b750-4626-8f15-dfbc009b647f-tls-certs\") pod \"e3aa2858-b750-4626-8f15-dfbc009b647f\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " Oct 30 22:24:23.581417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581402 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-tmp-dir\") pod \"e3aa2858-b750-4626-8f15-dfbc009b647f\" (UID: \"e3aa2858-b750-4626-8f15-dfbc009b647f\") " Oct 30 22:24:23.581512 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581487 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-home" pod "e3aa2858-b750-4626-8f15-dfbc009b647f" (UID: "e3aa2858-b750-4626-8f15-dfbc009b647f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:23.581687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581659 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-tmp-dir" pod "e3aa2858-b750-4626-8f15-dfbc009b647f" (UID: "e3aa2858-b750-4626-8f15-dfbc009b647f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:23.581867 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581850 4859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-home\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:23.582030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581885 4859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-tmp-dir\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:23.582030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.581894 4859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-model-cache\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:23.583542 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.583518 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3aa2858-b750-4626-8f15-dfbc009b647f-tls-certs" pod "e3aa2858-b750-4626-8f15-dfbc009b647f" (UID: "e3aa2858-b750-4626-8f15-dfbc009b647f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:23.583613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.583554 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-dshm" pod "e3aa2858-b750-4626-8f15-dfbc009b647f" (UID: "e3aa2858-b750-4626-8f15-dfbc009b647f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:23.583613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.583586 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aa2858-b750-4626-8f15-dfbc009b647f-kube-api-access-kb8jm" pod "e3aa2858-b750-4626-8f15-dfbc009b647f" (UID: "e3aa2858-b750-4626-8f15-dfbc009b647f"). InnerVolumeSpecName "kube-api-access-kb8jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:23.652653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.652621 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:24:23.682723 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.682695 4859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3aa2858-b750-4626-8f15-dfbc009b647f-dshm\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:23.682836 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.682726 4859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3aa2858-b750-4626-8f15-dfbc009b647f-tls-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:23.682836 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:23.682740 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb8jm\" (UniqueName: \"kubernetes.io/projected/e3aa2858-b750-4626-8f15-dfbc009b647f-kube-api-access-kb8jm\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:24.447715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:24.447683 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" Oct 30 22:24:24.447715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:24.447689 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9" event={"ID":"e3aa2858-b750-4626-8f15-dfbc009b647f","Type":"ContainerDied","Data":"da707eebcce31ffda7510e9a5e921d15fcf90766ec403f7b6c73f44a395a6eb5"} Oct 30 22:24:24.448309 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:24.447745 4859 scope.go:122] "RemoveContainer" containerID="ce802e95ddb863a867d0fdc5558cd99042098403ae3e4dd299b4491b6a07f98f" Oct 30 22:24:24.618169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:24.618132 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9"] Oct 30 22:24:24.665540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:24.665509 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2edb49/ipp-route-2edb49-kserve-7df94fc89-xknd9"] Oct 30 22:24:24.744320 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:24.744238 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3aa2858-b750-4626-8f15-dfbc009b647f" path="/var/lib/kubelet/pods/e3aa2858-b750-4626-8f15-dfbc009b647f/volumes" Oct 30 22:24:39.882037 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:24:39.881994 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:24:40.855411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:40.855380 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv"] Oct 30 22:24:40.855612 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:40.855587 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" podUID="4a6e6824-4265-4963-ab7e-e150ee7bcf28" containerName="maas-api" containerID="cri-o://5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e" gracePeriod=30 Oct 30 22:24:41.171239 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.171207 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:24:41.232477 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.232437 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-metrics-certs\") pod \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " Oct 30 22:24:41.232682 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.232617 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-maas-api-tls\") pod \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " Oct 30 22:24:41.232682 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.232656 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6e6824-4265-4963-ab7e-e150ee7bcf28-kube-api-access-vxqhb\" (UniqueName: \"kubernetes.io/projected/4a6e6824-4265-4963-ab7e-e150ee7bcf28-kube-api-access-vxqhb\") pod \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " Oct 30 22:24:41.232793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.232707 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6e6824-4265-4963-ab7e-e150ee7bcf28-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6e6824-4265-4963-ab7e-e150ee7bcf28-openshift-service-ca\") pod \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\" (UID: \"4a6e6824-4265-4963-ab7e-e150ee7bcf28\") " Oct 30 22:24:41.233154 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.233122 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6e6824-4265-4963-ab7e-e150ee7bcf28-openshift-service-ca" pod "4a6e6824-4265-4963-ab7e-e150ee7bcf28" (UID: "4a6e6824-4265-4963-ab7e-e150ee7bcf28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:41.234542 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.234514 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-metrics-certs" pod "4a6e6824-4265-4963-ab7e-e150ee7bcf28" (UID: "4a6e6824-4265-4963-ab7e-e150ee7bcf28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:41.234683 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.234668 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-maas-api-tls" pod "4a6e6824-4265-4963-ab7e-e150ee7bcf28" (UID: "4a6e6824-4265-4963-ab7e-e150ee7bcf28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:41.234748 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.234726 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6e6824-4265-4963-ab7e-e150ee7bcf28-kube-api-access-vxqhb" pod "4a6e6824-4265-4963-ab7e-e150ee7bcf28" (UID: "4a6e6824-4265-4963-ab7e-e150ee7bcf28"). InnerVolumeSpecName "kube-api-access-vxqhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:41.334405 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.334363 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6e6824-4265-4963-ab7e-e150ee7bcf28-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:41.334405 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.334406 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:41.334667 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.334423 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e6824-4265-4963-ab7e-e150ee7bcf28-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:41.334667 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.334438 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxqhb\" (UniqueName: \"kubernetes.io/projected/4a6e6824-4265-4963-ab7e-e150ee7bcf28-kube-api-access-vxqhb\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:41.567913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.567824 4859 generic.go:356] "Generic (PLEG): container finished" podID="4a6e6824-4265-4963-ab7e-e150ee7bcf28" containerID="5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e" exitCode=0 Oct 30 22:24:41.567913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.567899 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" Oct 30 22:24:41.568137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.567912 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" event={"ID":"4a6e6824-4265-4963-ab7e-e150ee7bcf28","Type":"ContainerDied","Data":"5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e"} Oct 30 22:24:41.568137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.567994 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv" event={"ID":"4a6e6824-4265-4963-ab7e-e150ee7bcf28","Type":"ContainerDied","Data":"d093ca8a4d8ed0daf6fc40d78dbe9a08da7e215181b01fc39f4981117ef1b5f8"} Oct 30 22:24:41.568137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.568027 4859 scope.go:122] "RemoveContainer" containerID="5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e" Oct 30 22:24:41.579096 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.579022 4859 scope.go:122] "RemoveContainer" containerID="5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e" Oct 30 22:24:41.579463 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:24:41.579430 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e\": container with ID starting with 5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e not found: ID does not exist" containerID="5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e" Oct 30 22:24:41.579632 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.579600 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e"} err="failed to get container status \"5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e\": rpc error: code = NotFound desc = could not find container \"5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e\": container with ID starting with 5e9f893ac82006f85f2cacd823a7efbf5be70417f060891c12a093676cb2241e not found: ID does not exist" Oct 30 22:24:41.732885 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.732847 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv"] Oct 30 22:24:41.735833 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:41.735811 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2565-67d588fc6c-db9tv"] Oct 30 22:24:42.742831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:42.742794 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6e6824-4265-4963-ab7e-e150ee7bcf28" path="/var/lib/kubelet/pods/4a6e6824-4265-4963-ab7e-e150ee7bcf28/volumes" Oct 30 22:24:49.950701 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:49.950667 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp"] Oct 30 22:24:49.951185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:49.950882 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" containerID="cri-o://ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5" gracePeriod=30 Oct 30 22:24:52.290332 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:52.290292 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5"] Oct 30 22:24:52.291021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:52.290989 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" containerID="cri-o://781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8" gracePeriod=30 Oct 30 22:24:53.531235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:53.531184 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:24:53.531724 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:53.531328 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:24:54.700282 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:54.700247 4859 generic.go:356] "Generic (PLEG): container finished" podID="5d209f91-4095-47a2-866d-d58f89e198a2" containerID="bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3" exitCode=0 Oct 30 22:24:54.700693 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:54.700319 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" event={"ID":"5d209f91-4095-47a2-866d-d58f89e198a2","Type":"ContainerDied","Data":"bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3"} Oct 30 22:24:54.700744 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:54.700700 4859 scope.go:122] "RemoveContainer" containerID="bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3" Oct 30 22:24:54.702182 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:54.702157 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:24:54.702248 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:54.702222 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:24:54.702283 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:54.702253 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:24:55.278534 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.278178 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:24:55.393195 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.393153 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-istio-token\" (UniqueName: \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-istio-token\") pod \"78bb9890-62cb-4557-9871-469bffa798a9\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.393613 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-certs\") pod \"78bb9890-62cb-4557-9871-469bffa798a9\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.393742 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-kube-api-access-twlbp\" (UniqueName: \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-kube-api-access-twlbp\") pod \"78bb9890-62cb-4557-9871-469bffa798a9\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.393772 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-credential-socket\") pod \"78bb9890-62cb-4557-9871-469bffa798a9\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.393809 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78bb9890-62cb-4557-9871-469bffa798a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78bb9890-62cb-4557-9871-469bffa798a9-istiod-ca-cert\") pod \"78bb9890-62cb-4557-9871-469bffa798a9\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.393935 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78bb9890-62cb-4557-9871-469bffa798a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78bb9890-62cb-4557-9871-469bffa798a9-istio-podinfo\") pod \"78bb9890-62cb-4557-9871-469bffa798a9\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.393982 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-socket\") pod \"78bb9890-62cb-4557-9871-469bffa798a9\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.394119 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-data\") pod \"78bb9890-62cb-4557-9871-469bffa798a9\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.394146 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-envoy\") pod \"78bb9890-62cb-4557-9871-469bffa798a9\" (UID: \"78bb9890-62cb-4557-9871-469bffa798a9\") " Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.394919 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-credential-socket" pod "78bb9890-62cb-4557-9871-469bffa798a9" (UID: "78bb9890-62cb-4557-9871-469bffa798a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.395253 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78bb9890-62cb-4557-9871-469bffa798a9-istiod-ca-cert" pod "78bb9890-62cb-4557-9871-469bffa798a9" (UID: "78bb9890-62cb-4557-9871-469bffa798a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.395317 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-socket" pod "78bb9890-62cb-4557-9871-469bffa798a9" (UID: "78bb9890-62cb-4557-9871-469bffa798a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:55.395398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.395373 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-data" pod "78bb9890-62cb-4557-9871-469bffa798a9" (UID: "78bb9890-62cb-4557-9871-469bffa798a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:55.396258 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.396208 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-certs" pod "78bb9890-62cb-4557-9871-469bffa798a9" (UID: "78bb9890-62cb-4557-9871-469bffa798a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:55.399179 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.398804 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78bb9890-62cb-4557-9871-469bffa798a9-istio-podinfo" pod "78bb9890-62cb-4557-9871-469bffa798a9" (UID: "78bb9890-62cb-4557-9871-469bffa798a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:55.421462 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.411069 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-envoy" pod "78bb9890-62cb-4557-9871-469bffa798a9" (UID: "78bb9890-62cb-4557-9871-469bffa798a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:55.426229 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.425123 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-istio-token" pod "78bb9890-62cb-4557-9871-469bffa798a9" (UID: "78bb9890-62cb-4557-9871-469bffa798a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:55.426229 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.425210 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-kube-api-access-twlbp" pod "78bb9890-62cb-4557-9871-469bffa798a9" (UID: "78bb9890-62cb-4557-9871-469bffa798a9"). InnerVolumeSpecName "kube-api-access-twlbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:55.496136 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.496037 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.496136 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.496074 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.496136 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.496092 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.496136 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.496107 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.496136 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.496123 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twlbp\" (UniqueName: \"kubernetes.io/projected/78bb9890-62cb-4557-9871-469bffa798a9-kube-api-access-twlbp\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.496136 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.496139 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.496590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.496153 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78bb9890-62cb-4557-9871-469bffa798a9-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.496590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.496171 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78bb9890-62cb-4557-9871-469bffa798a9-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.496590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.496185 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bb9890-62cb-4557-9871-469bffa798a9-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.708464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.708421 4859 generic.go:356] "Generic (PLEG): container finished" podID="78bb9890-62cb-4557-9871-469bffa798a9" containerID="ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5" exitCode=0 Oct 30 22:24:55.708898 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.708487 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" Oct 30 22:24:55.708898 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.708506 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" event={"ID":"78bb9890-62cb-4557-9871-469bffa798a9","Type":"ContainerDied","Data":"ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5"} Oct 30 22:24:55.708898 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.708563 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp" event={"ID":"78bb9890-62cb-4557-9871-469bffa798a9","Type":"ContainerDied","Data":"a3d683fce28f34f4b724a44353a990a20a34e4b9cd980da2cb32f05649a4b7ed"} Oct 30 22:24:55.708898 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.708586 4859 scope.go:122] "RemoveContainer" containerID="ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5" Oct 30 22:24:55.710977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.710934 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" event={"ID":"5d209f91-4095-47a2-866d-d58f89e198a2","Type":"ContainerStarted","Data":"c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103"} Oct 30 22:24:55.719696 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.719682 4859 scope.go:122] "RemoveContainer" containerID="eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5" Oct 30 22:24:55.728594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.728575 4859 scope.go:122] "RemoveContainer" containerID="ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5" Oct 30 22:24:55.728852 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:24:55.728831 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5\": container with ID starting with ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5 not found: ID does not exist" containerID="ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5" Oct 30 22:24:55.728918 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.728859 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5"} err="failed to get container status \"ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5\": rpc error: code = NotFound desc = could not find container \"ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5\": container with ID starting with ab916d470e8fc5b2114e5cd1ab3630f870a6a262a80e7b1f44c2724c97d897a5 not found: ID does not exist" Oct 30 22:24:55.728918 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.728875 4859 scope.go:122] "RemoveContainer" containerID="eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5" Oct 30 22:24:55.729171 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:24:55.729153 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5\": container with ID starting with eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5 not found: ID does not exist" containerID="eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5" Oct 30 22:24:55.729226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.729175 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5"} err="failed to get container status \"eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5\": rpc error: code = NotFound desc = could not find container \"eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5\": container with ID starting with eccf4fd65d9dd0086565f5a86692749fa17e10201a91861391d1cc3f8ee58ca5 not found: ID does not exist" Oct 30 22:24:55.891778 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.891745 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp"] Oct 30 22:24:55.939119 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:55.939080 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2565-openshift-default-896f55d6f-qmnxp"] Oct 30 22:24:56.609093 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:56.609062 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:24:56.609431 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:56.609401 4859 patch_prober.go:29] interesting pod/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:24:56.609696 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:56.609649 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 22:24:56.741688 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:56.741657 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78bb9890-62cb-4557-9871-469bffa798a9" path="/var/lib/kubelet/pods/78bb9890-62cb-4557-9871-469bffa798a9/volumes" Oct 30 22:24:56.782969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:56.782043 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:24:57.289969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.288099 4859 patch_prober.go:29] interesting pod/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:24:57.289969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.288172 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:24:57.607897 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.607868 4859 patch_prober.go:29] interesting pod/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:24:57.608013 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.607916 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 22:24:57.622104 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.622080 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:24:57.627195 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627166 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-certs\") pod \"9d60d248-a783-420e-a309-16db1b7ee00e\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " Oct 30 22:24:57.627304 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627220 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-kube-api-access-g4hm9\" (UniqueName: \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-kube-api-access-g4hm9\") pod \"9d60d248-a783-420e-a309-16db1b7ee00e\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " Oct 30 22:24:57.627304 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627252 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-istio-token\" (UniqueName: \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-istio-token\") pod \"9d60d248-a783-420e-a309-16db1b7ee00e\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " Oct 30 22:24:57.627414 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627302 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-socket\") pod \"9d60d248-a783-420e-a309-16db1b7ee00e\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " Oct 30 22:24:57.627414 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627347 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-credential-socket\") pod \"9d60d248-a783-420e-a309-16db1b7ee00e\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " Oct 30 22:24:57.627414 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627406 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-envoy\") pod \"9d60d248-a783-420e-a309-16db1b7ee00e\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " Oct 30 22:24:57.627571 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627422 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-certs" pod "9d60d248-a783-420e-a309-16db1b7ee00e" (UID: "9d60d248-a783-420e-a309-16db1b7ee00e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:57.627571 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627460 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d60d248-a783-420e-a309-16db1b7ee00e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60d248-a783-420e-a309-16db1b7ee00e-istiod-ca-cert\") pod \"9d60d248-a783-420e-a309-16db1b7ee00e\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " Oct 30 22:24:57.627571 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627512 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-data\") pod \"9d60d248-a783-420e-a309-16db1b7ee00e\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " Oct 30 22:24:57.627718 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627622 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d60d248-a783-420e-a309-16db1b7ee00e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60d248-a783-420e-a309-16db1b7ee00e-istio-podinfo\") pod \"9d60d248-a783-420e-a309-16db1b7ee00e\" (UID: \"9d60d248-a783-420e-a309-16db1b7ee00e\") " Oct 30 22:24:57.627718 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627639 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-socket" pod "9d60d248-a783-420e-a309-16db1b7ee00e" (UID: "9d60d248-a783-420e-a309-16db1b7ee00e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:57.627718 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627647 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-credential-socket" pod "9d60d248-a783-420e-a309-16db1b7ee00e" (UID: "9d60d248-a783-420e-a309-16db1b7ee00e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:57.627970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627840 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-data" pod "9d60d248-a783-420e-a309-16db1b7ee00e" (UID: "9d60d248-a783-420e-a309-16db1b7ee00e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:57.627970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.627850 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d60d248-a783-420e-a309-16db1b7ee00e-istiod-ca-cert" pod "9d60d248-a783-420e-a309-16db1b7ee00e" (UID: "9d60d248-a783-420e-a309-16db1b7ee00e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:57.628215 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.628189 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60d248-a783-420e-a309-16db1b7ee00e-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:57.628336 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.628223 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:57.628336 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.628238 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:57.628336 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.628254 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:57.628336 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.628269 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:57.629436 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.629415 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-kube-api-access-g4hm9" pod "9d60d248-a783-420e-a309-16db1b7ee00e" (UID: "9d60d248-a783-420e-a309-16db1b7ee00e"). InnerVolumeSpecName "kube-api-access-g4hm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:57.629600 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.629576 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d60d248-a783-420e-a309-16db1b7ee00e-istio-podinfo" pod "9d60d248-a783-420e-a309-16db1b7ee00e" (UID: "9d60d248-a783-420e-a309-16db1b7ee00e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:57.629600 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.629589 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-istio-token" pod "9d60d248-a783-420e-a309-16db1b7ee00e" (UID: "9d60d248-a783-420e-a309-16db1b7ee00e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:57.629929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.629915 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-envoy" pod "9d60d248-a783-420e-a309-16db1b7ee00e" (UID: "9d60d248-a783-420e-a309-16db1b7ee00e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:57.729114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.729083 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60d248-a783-420e-a309-16db1b7ee00e-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:57.729114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.729114 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4hm9\" (UniqueName: \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-kube-api-access-g4hm9\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:57.729365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.729126 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60d248-a783-420e-a309-16db1b7ee00e-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:57.729365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.729136 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60d248-a783-420e-a309-16db1b7ee00e-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:57.729683 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.729656 4859 generic.go:356] "Generic (PLEG): container finished" podID="9d60d248-a783-420e-a309-16db1b7ee00e" containerID="781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8" exitCode=0 Oct 30 22:24:57.729752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.729703 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" Oct 30 22:24:57.729752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.729736 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" event={"ID":"9d60d248-a783-420e-a309-16db1b7ee00e","Type":"ContainerDied","Data":"781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8"} Oct 30 22:24:57.729817 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.729791 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5" event={"ID":"9d60d248-a783-420e-a309-16db1b7ee00e","Type":"ContainerDied","Data":"2ceefda054f3eaf74ad92a7346f4dbb59f2346a9aef746dc6dd06e8cf2336f98"} Oct 30 22:24:57.729849 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.729818 4859 scope.go:122] "RemoveContainer" containerID="781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8" Oct 30 22:24:57.741237 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.741215 4859 scope.go:122] "RemoveContainer" containerID="a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7" Oct 30 22:24:57.750436 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.750407 4859 scope.go:122] "RemoveContainer" containerID="781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8" Oct 30 22:24:57.750729 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:24:57.750683 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8\": container with ID starting with 781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8 not found: ID does not exist" containerID="781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8" Oct 30 22:24:57.750729 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.750705 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8"} err="failed to get container status \"781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8\": rpc error: code = NotFound desc = could not find container \"781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8\": container with ID starting with 781788800c1e7a2eb1f70e944874ec564f2ae042c41e6e507e0ed5f615b87db8 not found: ID does not exist" Oct 30 22:24:57.750729 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.750722 4859 scope.go:122] "RemoveContainer" containerID="a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7" Oct 30 22:24:57.750965 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:24:57.750922 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7\": container with ID starting with a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7 not found: ID does not exist" containerID="a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7" Oct 30 22:24:57.751018 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.750960 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7"} err="failed to get container status \"a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7\": rpc error: code = NotFound desc = could not find container \"a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7\": container with ID starting with a49ff82f2d716230892dc2e2cb45b87edd361b9ff3e3c24cd1d84f62a24460b7 not found: ID does not exist" Oct 30 22:24:57.936106 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.936022 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5"] Oct 30 22:24:57.940205 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:57.940178 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2edb49-openshift-default-7fdb57977f-xzlb5"] Oct 30 22:24:58.608821 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:58.608788 4859 patch_prober.go:29] interesting pod/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:24:58.609035 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:58.608870 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 22:24:58.740793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:58.740761 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" path="/var/lib/kubelet/pods/9d60d248-a783-420e-a309-16db1b7ee00e/volumes" Oct 30 22:24:59.608372 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:59.608339 4859 patch_prober.go:29] interesting pod/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:24:59.608791 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:24:59.608394 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 22:25:00.608488 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:00.608448 4859 patch_prober.go:29] interesting pod/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:25:00.608887 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:00.608509 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 22:25:01.612986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:01.612934 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:25:01.614026 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:01.614003 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:25:09.884556 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:25:09.884363 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:25:11.037224 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:11.037182 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:25:14.683705 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:14.683667 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz"] Oct 30 22:25:14.684143 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:14.683899 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" podUID="c8a42bc5-a366-43fa-b540-3ad76aee16bb" containerName="maas-api" containerID="cri-o://e1117c30da071cacf4b4623353c41d7f752fde2181568886a6d03881ebdd2b13" gracePeriod=30 Oct 30 22:25:14.847907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:14.847870 4859 generic.go:356] "Generic (PLEG): container finished" podID="c8a42bc5-a366-43fa-b540-3ad76aee16bb" containerID="e1117c30da071cacf4b4623353c41d7f752fde2181568886a6d03881ebdd2b13" exitCode=0 Oct 30 22:25:14.848113 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:14.848094 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" event={"ID":"c8a42bc5-a366-43fa-b540-3ad76aee16bb","Type":"ContainerDied","Data":"e1117c30da071cacf4b4623353c41d7f752fde2181568886a6d03881ebdd2b13"} Oct 30 22:25:14.988337 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:14.988313 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:25:15.156559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.156522 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-maas-api-tls\") pod \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " Oct 30 22:25:15.156732 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.156625 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-metrics-certs\") pod \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " Oct 30 22:25:15.156732 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.156673 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a42bc5-a366-43fa-b540-3ad76aee16bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a42bc5-a366-43fa-b540-3ad76aee16bb-openshift-service-ca\") pod \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " Oct 30 22:25:15.156732 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.156707 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a42bc5-a366-43fa-b540-3ad76aee16bb-kube-api-access-bhcjx\" (UniqueName: \"kubernetes.io/projected/c8a42bc5-a366-43fa-b540-3ad76aee16bb-kube-api-access-bhcjx\") pod \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\" (UID: \"c8a42bc5-a366-43fa-b540-3ad76aee16bb\") " Oct 30 22:25:15.157131 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.157108 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a42bc5-a366-43fa-b540-3ad76aee16bb-openshift-service-ca" pod "c8a42bc5-a366-43fa-b540-3ad76aee16bb" (UID: "c8a42bc5-a366-43fa-b540-3ad76aee16bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:15.158857 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.158831 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a42bc5-a366-43fa-b540-3ad76aee16bb-kube-api-access-bhcjx" pod "c8a42bc5-a366-43fa-b540-3ad76aee16bb" (UID: "c8a42bc5-a366-43fa-b540-3ad76aee16bb"). InnerVolumeSpecName "kube-api-access-bhcjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:15.159234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.159211 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-metrics-certs" pod "c8a42bc5-a366-43fa-b540-3ad76aee16bb" (UID: "c8a42bc5-a366-43fa-b540-3ad76aee16bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:15.159234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.159227 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-maas-api-tls" pod "c8a42bc5-a366-43fa-b540-3ad76aee16bb" (UID: "c8a42bc5-a366-43fa-b540-3ad76aee16bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:15.262710 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.262622 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:15.262914 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.262897 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a42bc5-a366-43fa-b540-3ad76aee16bb-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:15.263024 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.263010 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhcjx\" (UniqueName: \"kubernetes.io/projected/c8a42bc5-a366-43fa-b540-3ad76aee16bb-kube-api-access-bhcjx\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:15.263106 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.263095 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a42bc5-a366-43fa-b540-3ad76aee16bb-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:15.863530 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.863492 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" event={"ID":"c8a42bc5-a366-43fa-b540-3ad76aee16bb","Type":"ContainerDied","Data":"739f3305e0a9e433fd5693c46573a56d9e56d32b5f1f810df1c1fb0ff5e3b5c5"} Oct 30 22:25:15.864020 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.863568 4859 scope.go:122] "RemoveContainer" containerID="e1117c30da071cacf4b4623353c41d7f752fde2181568886a6d03881ebdd2b13" Oct 30 22:25:15.864020 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:15.863740 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz" Oct 30 22:25:16.032338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:16.032299 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz"] Oct 30 22:25:16.081694 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:16.081655 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96567a-6c99c9cb5d-jptnz"] Oct 30 22:25:16.741095 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:16.741060 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a42bc5-a366-43fa-b540-3ad76aee16bb" path="/var/lib/kubelet/pods/c8a42bc5-a366-43fa-b540-3ad76aee16bb/volumes" Oct 30 22:25:39.876571 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:25:39.876527 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:25:51.927092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:25:51.927057 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:26:09.885233 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:26:09.885192 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:26:21.527973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:21.527927 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:26:25.605107 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605073 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff"] Oct 30 22:26:25.605604 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605589 4859 cpu_manager.go:414] "containerMap: removing container" podUID="c8a42bc5-a366-43fa-b540-3ad76aee16bb" containerName="maas-api" Oct 30 22:26:25.605649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605605 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a42bc5-a366-43fa-b540-3ad76aee16bb" containerName="maas-api" Oct 30 22:26:25.605649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605619 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f454daf2-9772-4eba-81fc-d9acf70939ea" containerName="maas-api" Oct 30 22:26:25.605649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605624 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f454daf2-9772-4eba-81fc-d9acf70939ea" containerName="maas-api" Oct 30 22:26:25.605649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605633 4859 cpu_manager.go:414] "containerMap: removing container" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerName="istio-proxy" Oct 30 22:26:25.605649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605638 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerName="istio-proxy" Oct 30 22:26:25.605649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605649 4859 cpu_manager.go:414] "containerMap: removing container" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605655 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605666 4859 cpu_manager.go:414] "containerMap: removing container" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605670 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605676 4859 cpu_manager.go:414] "containerMap: removing container" podUID="4a6e6824-4265-4963-ab7e-e150ee7bcf28" containerName="maas-api" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605682 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6e6824-4265-4963-ab7e-e150ee7bcf28" containerName="maas-api" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605692 4859 cpu_manager.go:414] "containerMap: removing container" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605696 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605718 4859 cpu_manager.go:414] "containerMap: removing container" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605722 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605729 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e3aa2858-b750-4626-8f15-dfbc009b647f" containerName="main" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605734 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3aa2858-b750-4626-8f15-dfbc009b647f" containerName="main" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605805 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605812 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605818 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3aa2858-b750-4626-8f15-dfbc009b647f" containerName="main" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605826 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a42bc5-a366-43fa-b540-3ad76aee16bb" containerName="maas-api" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605834 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f454daf2-9772-4eba-81fc-d9acf70939ea" containerName="maas-api" Oct 30 22:26:25.605835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605840 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77f54c3-1e05-4549-bdd9-d2a5a768d84f" containerName="istio-proxy" Oct 30 22:26:25.606380 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605847 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="571531de-dec4-474a-b197-8d84823a8198" containerName="istio-proxy" Oct 30 22:26:25.606380 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.605852 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6e6824-4265-4963-ab7e-e150ee7bcf28" containerName="maas-api" Oct 30 22:26:25.609368 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.609348 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.698219 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.698181 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.698415 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.698296 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6692p\" (UniqueName: \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-kube-api-access-6692p\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.698415 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.698364 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.698515 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.698466 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.698568 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.698520 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.698610 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.698596 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.698656 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.698619 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.698709 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.698698 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.698750 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.698721 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800140 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800095 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800330 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800153 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800330 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800212 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800330 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800267 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6692p\" (UniqueName: \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-kube-api-access-6692p\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800330 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800320 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800369 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800411 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800509 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800519 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800543 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.800961 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.800902 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.801191 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.801164 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.801282 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.801217 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.801483 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.801461 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.802590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.802569 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.803175 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.803153 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.843539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.843498 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff"] Oct 30 22:26:25.861256 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.861186 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.861378 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.861271 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6692p\" (UniqueName: \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-kube-api-access-6692p\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:25.923270 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:25.923243 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:26.374465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:26.374436 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff"] Oct 30 22:26:26.375418 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:26:26.375395 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76a03cc_7bd2_4fb9_a6a7_1df7f9577acd.slice/crio-253bdddc5213ef2f51767e6f20c01f501f311acf2f42d8e370d730d8b67cf274 WatchSource:0}: Error finding container 253bdddc5213ef2f51767e6f20c01f501f311acf2f42d8e370d730d8b67cf274: Status 404 returned error can't find the container with id 253bdddc5213ef2f51767e6f20c01f501f311acf2f42d8e370d730d8b67cf274 Oct 30 22:26:26.377705 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:26.377671 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:26:26.377774 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:26.377738 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:26:26.377774 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:26.377764 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:26:27.395451 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:27.395415 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" event={"ID":"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd","Type":"ContainerStarted","Data":"4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03"} Oct 30 22:26:27.395861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:27.395469 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" event={"ID":"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd","Type":"ContainerStarted","Data":"253bdddc5213ef2f51767e6f20c01f501f311acf2f42d8e370d730d8b67cf274"} Oct 30 22:26:27.614304 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:27.614247 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" podStartSLOduration=2.614232343 podStartE2EDuration="2.614232343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:27.564292422 +0000 UTC m=+2644.936919193" watchObservedRunningTime="2026-10-30 22:26:27.614232343 +0000 UTC m=+2644.986859090" Oct 30 22:26:27.924593 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:27.924556 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:27.926192 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:27.926166 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:26:27.926291 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:27.926222 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:26:28.923738 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:28.923702 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:26:28.924175 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:28.923764 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:26:29.924301 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:29.924263 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:26:29.924802 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:29.924334 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:26:30.923994 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:30.923935 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:26:30.924207 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:30.924019 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:26:31.923733 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:31.923694 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:26:31.924168 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:31.923769 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:26:32.924419 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:32.924388 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:26:32.924814 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:32.924437 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:26:33.928265 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:33.928236 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:33.928696 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:33.928683 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:33.929467 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:33.929443 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:26:35.655114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:35.655027 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6"] Oct 30 22:26:35.655540 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:35.655339 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" podUID="6ac0b05b-f073-4643-ab5d-6771be17eeb1" containerName="maas-api" containerID="cri-o://2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63" gracePeriod=30 Oct 30 22:26:35.975263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:35.975232 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:26:36.052681 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.052638 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac0b05b-f073-4643-ab5d-6771be17eeb1-kube-api-access-96vzw\" (UniqueName: \"kubernetes.io/projected/6ac0b05b-f073-4643-ab5d-6771be17eeb1-kube-api-access-96vzw\") pod \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " Oct 30 22:26:36.063579 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.052932 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-maas-api-tls\") pod \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " Oct 30 22:26:36.063579 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.053073 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-metrics-certs\") pod \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " Oct 30 22:26:36.063579 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.053212 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac0b05b-f073-4643-ab5d-6771be17eeb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0b05b-f073-4643-ab5d-6771be17eeb1-openshift-service-ca\") pod \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\" (UID: \"6ac0b05b-f073-4643-ab5d-6771be17eeb1\") " Oct 30 22:26:36.064119 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.064085 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac0b05b-f073-4643-ab5d-6771be17eeb1-openshift-service-ca" pod "6ac0b05b-f073-4643-ab5d-6771be17eeb1" (UID: "6ac0b05b-f073-4643-ab5d-6771be17eeb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:36.088263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.088224 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-maas-api-tls" pod "6ac0b05b-f073-4643-ab5d-6771be17eeb1" (UID: "6ac0b05b-f073-4643-ab5d-6771be17eeb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:36.089160 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.089119 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac0b05b-f073-4643-ab5d-6771be17eeb1-kube-api-access-96vzw" pod "6ac0b05b-f073-4643-ab5d-6771be17eeb1" (UID: "6ac0b05b-f073-4643-ab5d-6771be17eeb1"). InnerVolumeSpecName "kube-api-access-96vzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:36.089390 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.089373 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-metrics-certs" pod "6ac0b05b-f073-4643-ab5d-6771be17eeb1" (UID: "6ac0b05b-f073-4643-ab5d-6771be17eeb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:36.155149 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.155111 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0b05b-f073-4643-ab5d-6771be17eeb1-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:36.155149 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.155144 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96vzw\" (UniqueName: \"kubernetes.io/projected/6ac0b05b-f073-4643-ab5d-6771be17eeb1-kube-api-access-96vzw\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:36.155388 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.155158 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:36.155388 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.155177 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac0b05b-f073-4643-ab5d-6771be17eeb1-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:36.460819 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.460780 4859 generic.go:356] "Generic (PLEG): container finished" podID="6ac0b05b-f073-4643-ab5d-6771be17eeb1" containerID="2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63" exitCode=0 Oct 30 22:26:36.461041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.460830 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" Oct 30 22:26:36.461041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.460865 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" event={"ID":"6ac0b05b-f073-4643-ab5d-6771be17eeb1","Type":"ContainerDied","Data":"2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63"} Oct 30 22:26:36.461041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.460936 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6" event={"ID":"6ac0b05b-f073-4643-ab5d-6771be17eeb1","Type":"ContainerDied","Data":"5392db00a18ddcbac8c35bd68e70c57e41c4cef0f7c6b36f6c5c18e2e67e5723"} Oct 30 22:26:36.461041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.460997 4859 scope.go:122] "RemoveContainer" containerID="2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63" Oct 30 22:26:36.473939 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.473918 4859 scope.go:122] "RemoveContainer" containerID="2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63" Oct 30 22:26:36.474258 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:26:36.474240 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63\": container with ID starting with 2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63 not found: ID does not exist" containerID="2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63" Oct 30 22:26:36.474312 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.474267 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63"} err="failed to get container status \"2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63\": rpc error: code = NotFound desc = could not find container \"2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63\": container with ID starting with 2d4bf186cb8980a1fde1e36da3e8d187d96362fd4989a7cb1fd4452aa4c3ee63 not found: ID does not exist" Oct 30 22:26:36.624771 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.624739 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6"] Oct 30 22:26:36.627633 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.627609 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-955887-67d68886d8-2dxr6"] Oct 30 22:26:36.758759 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:36.758680 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac0b05b-f073-4643-ab5d-6771be17eeb1" path="/var/lib/kubelet/pods/6ac0b05b-f073-4643-ab5d-6771be17eeb1/volumes" Oct 30 22:26:39.878062 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:26:39.878027 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:26:42.691204 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.691158 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n"] Oct 30 22:26:42.691738 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.691723 4859 cpu_manager.go:414] "containerMap: removing container" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" Oct 30 22:26:42.691786 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.691739 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" Oct 30 22:26:42.691786 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.691762 4859 cpu_manager.go:414] "containerMap: removing container" podUID="6ac0b05b-f073-4643-ab5d-6771be17eeb1" containerName="maas-api" Oct 30 22:26:42.691786 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.691767 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac0b05b-f073-4643-ab5d-6771be17eeb1" containerName="maas-api" Oct 30 22:26:42.691884 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.691855 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac0b05b-f073-4643-ab5d-6771be17eeb1" containerName="maas-api" Oct 30 22:26:42.691884 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.691865 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d60d248-a783-420e-a309-16db1b7ee00e" containerName="istio-proxy" Oct 30 22:26:42.691884 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.691876 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="78bb9890-62cb-4557-9871-469bffa798a9" containerName="istio-proxy" Oct 30 22:26:42.696819 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.696799 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.767870 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.767827 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdd38ed-b9be-4835-9b42-1e508a764574-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.768097 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.767939 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbw9t\" (UniqueName: \"kubernetes.io/projected/9cdd38ed-b9be-4835-9b42-1e508a764574-kube-api-access-sbw9t\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.768097 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.768059 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.768217 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.768182 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-metrics-certs\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.798575 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.798542 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-5b1e39-serving-cert\"" Oct 30 22:26:42.801970 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.801931 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n"] Oct 30 22:26:42.868862 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.868825 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdd38ed-b9be-4835-9b42-1e508a764574-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.869078 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.868868 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbw9t\" (UniqueName: \"kubernetes.io/projected/9cdd38ed-b9be-4835-9b42-1e508a764574-kube-api-access-sbw9t\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.869078 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.868929 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.869216 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:26:42.869084 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-serving-cert: secret "maas-api-e2e-praxis-b-5b1e39-serving-cert" not found Oct 30 22:26:42.869216 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:26:42.869145 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-maas-api-tls podName:9cdd38ed-b9be-4835-9b42-1e508a764574 nodeName:}" failed. No retries permitted until 2026-10-30 22:26:43.369124965 +0000 UTC m=+2660.741751691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-maas-api-tls") pod "maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" (UID: "9cdd38ed-b9be-4835-9b42-1e508a764574") : secret "maas-api-e2e-praxis-b-5b1e39-serving-cert" not found Oct 30 22:26:42.869216 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.869173 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-metrics-certs\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.869546 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.869525 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdd38ed-b9be-4835-9b42-1e508a764574-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.871550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.871531 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-metrics-certs\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:42.950300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:42.950185 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbw9t\" (UniqueName: \"kubernetes.io/projected/9cdd38ed-b9be-4835-9b42-1e508a764574-kube-api-access-sbw9t\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:43.375863 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:43.375823 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:43.378262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:43.378236 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:43.610871 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:43.610831 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:43.995342 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:43.995311 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n"] Oct 30 22:26:43.996320 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:26:43.996294 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdd38ed_b9be_4835_9b42_1e508a764574.slice/crio-c346138e7d9e9325d778cd960b483b31e57b68ad56ca79fb75f010ff9ecbe178 WatchSource:0}: Error finding container c346138e7d9e9325d778cd960b483b31e57b68ad56ca79fb75f010ff9ecbe178: Status 404 returned error can't find the container with id c346138e7d9e9325d778cd960b483b31e57b68ad56ca79fb75f010ff9ecbe178 Oct 30 22:26:43.997769 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:43.997751 4859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:26:44.552584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:44.552543 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" event={"ID":"9cdd38ed-b9be-4835-9b42-1e508a764574","Type":"ContainerStarted","Data":"c346138e7d9e9325d778cd960b483b31e57b68ad56ca79fb75f010ff9ecbe178"} Oct 30 22:26:45.035131 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:45.035088 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4"] Oct 30 22:26:45.035607 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:45.035324 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" containerID="cri-o://4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4" gracePeriod=30 Oct 30 22:26:45.561438 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:45.561404 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" event={"ID":"9cdd38ed-b9be-4835-9b42-1e508a764574","Type":"ContainerStarted","Data":"dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f"} Oct 30 22:26:45.561619 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:45.561517 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:45.678711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:45.678659 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" podStartSLOduration=2.829966305 podStartE2EDuration="3.678644211s" totalImagesPullingTime="848.677906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:45.675442387 +0000 UTC m=+2663.048069131" watchObservedRunningTime="2026-10-30 22:26:45.678644211 +0000 UTC m=+2663.051270952" Oct 30 22:26:50.333416 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.333387 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:26:50.466876 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.466843 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-kube-api-access-8qxbq\" (UniqueName: \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-kube-api-access-8qxbq\") pod \"828e5250-0bee-44d5-a646-96e08952ccd3\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " Oct 30 22:26:50.467094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.466891 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-istio-token\" (UniqueName: \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-istio-token\") pod \"828e5250-0bee-44d5-a646-96e08952ccd3\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " Oct 30 22:26:50.467094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.466938 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-socket\") pod \"828e5250-0bee-44d5-a646-96e08952ccd3\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " Oct 30 22:26:50.467094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467008 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/828e5250-0bee-44d5-a646-96e08952ccd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828e5250-0bee-44d5-a646-96e08952ccd3-istiod-ca-cert\") pod \"828e5250-0bee-44d5-a646-96e08952ccd3\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " Oct 30 22:26:50.467094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467038 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-data\") pod \"828e5250-0bee-44d5-a646-96e08952ccd3\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " Oct 30 22:26:50.467315 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467147 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-envoy\") pod \"828e5250-0bee-44d5-a646-96e08952ccd3\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " Oct 30 22:26:50.467315 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467234 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-credential-socket\") pod \"828e5250-0bee-44d5-a646-96e08952ccd3\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " Oct 30 22:26:50.467315 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467282 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-certs\") pod \"828e5250-0bee-44d5-a646-96e08952ccd3\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " Oct 30 22:26:50.467475 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467301 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-socket" pod "828e5250-0bee-44d5-a646-96e08952ccd3" (UID: "828e5250-0bee-44d5-a646-96e08952ccd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:50.467532 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467485 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/828e5250-0bee-44d5-a646-96e08952ccd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828e5250-0bee-44d5-a646-96e08952ccd3-istio-podinfo\") pod \"828e5250-0bee-44d5-a646-96e08952ccd3\" (UID: \"828e5250-0bee-44d5-a646-96e08952ccd3\") " Oct 30 22:26:50.467598 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467570 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-certs" pod "828e5250-0bee-44d5-a646-96e08952ccd3" (UID: "828e5250-0bee-44d5-a646-96e08952ccd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:50.467598 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467580 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-credential-socket" pod "828e5250-0bee-44d5-a646-96e08952ccd3" (UID: "828e5250-0bee-44d5-a646-96e08952ccd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:50.467754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467716 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-data" pod "828e5250-0bee-44d5-a646-96e08952ccd3" (UID: "828e5250-0bee-44d5-a646-96e08952ccd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:50.468092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.467997 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:50.468092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.468025 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:50.468092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.468040 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:50.468092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.468054 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:50.468092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.468056 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828e5250-0bee-44d5-a646-96e08952ccd3-istiod-ca-cert" pod "828e5250-0bee-44d5-a646-96e08952ccd3" (UID: "828e5250-0bee-44d5-a646-96e08952ccd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:50.469648 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.469614 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-kube-api-access-8qxbq" pod "828e5250-0bee-44d5-a646-96e08952ccd3" (UID: "828e5250-0bee-44d5-a646-96e08952ccd3"). InnerVolumeSpecName "kube-api-access-8qxbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:50.469995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.469965 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/828e5250-0bee-44d5-a646-96e08952ccd3-istio-podinfo" pod "828e5250-0bee-44d5-a646-96e08952ccd3" (UID: "828e5250-0bee-44d5-a646-96e08952ccd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:50.470099 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.470014 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-istio-token" pod "828e5250-0bee-44d5-a646-96e08952ccd3" (UID: "828e5250-0bee-44d5-a646-96e08952ccd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:50.470234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.470210 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-envoy" pod "828e5250-0bee-44d5-a646-96e08952ccd3" (UID: "828e5250-0bee-44d5-a646-96e08952ccd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:50.568808 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.568708 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828e5250-0bee-44d5-a646-96e08952ccd3-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:50.568808 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.568744 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxbq\" (UniqueName: \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-kube-api-access-8qxbq\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:50.568808 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.568760 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828e5250-0bee-44d5-a646-96e08952ccd3-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:50.568808 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.568772 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828e5250-0bee-44d5-a646-96e08952ccd3-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:50.568808 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.568785 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828e5250-0bee-44d5-a646-96e08952ccd3-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:50.597513 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.597471 4859 generic.go:356] "Generic (PLEG): container finished" podID="828e5250-0bee-44d5-a646-96e08952ccd3" containerID="4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4" exitCode=0 Oct 30 22:26:50.597695 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.597675 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" event={"ID":"828e5250-0bee-44d5-a646-96e08952ccd3","Type":"ContainerDied","Data":"4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4"} Oct 30 22:26:50.597763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.597733 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" event={"ID":"828e5250-0bee-44d5-a646-96e08952ccd3","Type":"ContainerDied","Data":"aefac87e90d365a938b541cb9997754890e0e15191f13aaa7f3c14374598ad83"} Oct 30 22:26:50.597763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.597758 4859 scope.go:122] "RemoveContainer" containerID="4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4" Oct 30 22:26:50.597930 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.597898 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4" Oct 30 22:26:50.611706 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.611675 4859 scope.go:122] "RemoveContainer" containerID="92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb" Oct 30 22:26:50.622802 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.622776 4859 scope.go:122] "RemoveContainer" containerID="4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4" Oct 30 22:26:50.623158 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:26:50.623137 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4\": container with ID starting with 4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4 not found: ID does not exist" containerID="4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4" Oct 30 22:26:50.623263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.623179 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4"} err="failed to get container status \"4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4\": rpc error: code = NotFound desc = could not find container \"4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4\": container with ID starting with 4992f0e25946a591ed1158638d538c6d02d6668a2e9b08d8abe2e2946a8a8eb4 not found: ID does not exist" Oct 30 22:26:50.623263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.623198 4859 scope.go:122] "RemoveContainer" containerID="92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb" Oct 30 22:26:50.623494 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:26:50.623475 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb\": container with ID starting with 92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb not found: ID does not exist" containerID="92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb" Oct 30 22:26:50.623565 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.623498 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb"} err="failed to get container status \"92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb\": rpc error: code = NotFound desc = could not find container \"92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb\": container with ID starting with 92d0835d130be79826e1e12a473a167ece56604fc4a8ff722d78a34474f1a1eb not found: ID does not exist" Oct 30 22:26:50.764280 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.764204 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4"] Oct 30 22:26:50.767767 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:50.767739 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-955887-openshift-default-5df5db6df-tb8g4"] Oct 30 22:26:51.573828 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:51.573795 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:26:52.741982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:52.741930 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" path="/var/lib/kubelet/pods/828e5250-0bee-44d5-a646-96e08952ccd3/volumes" Oct 30 22:26:55.922899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:26:55.922856 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:27:09.882766 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:27:09.882722 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:27:17.210489 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.210297 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d"] Oct 30 22:27:17.223637 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.211623 4859 cpu_manager.go:414] "containerMap: removing container" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" Oct 30 22:27:17.223637 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.211645 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" Oct 30 22:27:17.223637 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.211876 4859 cpu_manager.go:414] "containerMap: removing container" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" Oct 30 22:27:17.223637 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.211896 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" Oct 30 22:27:17.223637 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.212073 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" Oct 30 22:27:17.238137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.238099 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.313223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.309556 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn7c5\" (UniqueName: \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-kube-api-access-bn7c5\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.313223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.309621 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e010b301-bf27-4059-ad1c-0e9e94d05684-istiod-ca-cert\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.313223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.309679 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-credential-socket\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.313223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.309730 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-socket\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.313223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.313028 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-podinfo\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.313640 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.313330 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-token\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.313640 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.313382 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-data\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.313640 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.313414 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-envoy\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.313640 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.313530 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-certs\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.314181 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.314146 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-42e8a1-openshift-default-dockercfg-8wn7r\"" Oct 30 22:27:17.332088 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.329643 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d"] Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.417579 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-token\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.417645 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-data\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.417681 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-envoy\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.417799 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-certs\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.417836 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7c5\" (UniqueName: \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-kube-api-access-bn7c5\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.417872 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e010b301-bf27-4059-ad1c-0e9e94d05684-istiod-ca-cert\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.417928 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-credential-socket\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.417995 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-socket\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.418023 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-podinfo\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.418104 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-data\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.418954 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-socket\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.419230 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-certs\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.419885 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e010b301-bf27-4059-ad1c-0e9e94d05684-istiod-ca-cert\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.420232 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-credential-socket\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.422641 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.421307 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-envoy\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.433840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.423400 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-podinfo\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.476466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.476395 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7c5\" (UniqueName: \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-kube-api-access-bn7c5\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.479170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.479130 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-token\") pod \"e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:17.579454 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:17.579417 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:18.063936 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:27:18.063905 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode010b301_bf27_4059_ad1c_0e9e94d05684.slice/crio-647e46b2e4f7c6ba34bac927fa5ba999a0e9738919730bfae451aa4d0fadb011 WatchSource:0}: Error finding container 647e46b2e4f7c6ba34bac927fa5ba999a0e9738919730bfae451aa4d0fadb011: Status 404 returned error can't find the container with id 647e46b2e4f7c6ba34bac927fa5ba999a0e9738919730bfae451aa4d0fadb011 Oct 30 22:27:18.064163 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:18.064136 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d"] Oct 30 22:27:18.066668 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:18.066630 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:18.066770 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:18.066711 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:18.066770 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:18.066754 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:18.793382 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:18.793333 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" event={"ID":"e010b301-bf27-4059-ad1c-0e9e94d05684","Type":"ContainerStarted","Data":"f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe"} Oct 30 22:27:18.793778 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:18.793409 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" event={"ID":"e010b301-bf27-4059-ad1c-0e9e94d05684","Type":"ContainerStarted","Data":"647e46b2e4f7c6ba34bac927fa5ba999a0e9738919730bfae451aa4d0fadb011"} Oct 30 22:27:18.962833 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:18.962775 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" podStartSLOduration=1.9627623189999999 podStartE2EDuration="1.962762319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:18.958857005 +0000 UTC m=+2696.331483751" watchObservedRunningTime="2026-10-30 22:27:18.962762319 +0000 UTC m=+2696.335389066" Oct 30 22:27:19.580326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:19.580283 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:19.586150 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:19.586122 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:19.800417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:19.800382 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:19.801349 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:19.801330 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:27:23.253047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:23.253019 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:27:23.261062 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:23.261036 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:27:23.268796 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:23.268767 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:27:23.277265 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:23.277235 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:27:25.657001 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.648193 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn"] Oct 30 22:27:25.677839 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.677780 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="828e5250-0bee-44d5-a646-96e08952ccd3" containerName="istio-proxy" Oct 30 22:27:25.689115 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.689076 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.771479 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.771436 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn"] Oct 30 22:27:25.778745 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.778710 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwjt\" (UniqueName: \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-kube-api-access-4pwjt\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.778745 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.778750 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.779047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.778794 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.779047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.778834 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.779047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.778871 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.779047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.778896 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.779047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.778988 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.779047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.779044 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95e7b7c-f119-4ead-9f6a-a170e854e421-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.779395 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.779073 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.879853 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.879810 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880106 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.879894 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880106 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.880006 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880106 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.880039 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880294 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.880109 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880294 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.880133 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95e7b7c-f119-4ead-9f6a-a170e854e421-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880294 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.880152 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880294 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.880181 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwjt\" (UniqueName: \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-kube-api-access-4pwjt\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880294 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.880199 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880294 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.880280 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880581 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.880555 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.880657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.880590 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.881427 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.881402 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95e7b7c-f119-4ead-9f6a-a170e854e421-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.881854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.881827 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.883578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.883482 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.883907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.883681 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.937704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.937631 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwjt\" (UniqueName: \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-kube-api-access-4pwjt\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:25.938586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:25.938566 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:26.043546 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:26.043502 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:26.488039 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:26.488003 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn"] Oct 30 22:27:26.488436 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:27:26.488410 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95e7b7c_f119_4ead_9f6a_a170e854e421.slice/crio-90d07a19e702f344be1c96cd2d0f2ad1d383447a1a228228b38c7c074f867bfa WatchSource:0}: Error finding container 90d07a19e702f344be1c96cd2d0f2ad1d383447a1a228228b38c7c074f867bfa: Status 404 returned error can't find the container with id 90d07a19e702f344be1c96cd2d0f2ad1d383447a1a228228b38c7c074f867bfa Oct 30 22:27:26.490782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:26.490748 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:26.490868 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:26.490812 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:26.490868 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:26.490840 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:26.853745 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:26.853695 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" event={"ID":"f95e7b7c-f119-4ead-9f6a-a170e854e421","Type":"ContainerStarted","Data":"8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844"} Oct 30 22:27:26.854201 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:26.853776 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" event={"ID":"f95e7b7c-f119-4ead-9f6a-a170e854e421","Type":"ContainerStarted","Data":"90d07a19e702f344be1c96cd2d0f2ad1d383447a1a228228b38c7c074f867bfa"} Oct 30 22:27:26.970115 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:26.969877 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podStartSLOduration=1.9698569620000002 podStartE2EDuration="1.969856962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:26.969087298 +0000 UTC m=+2704.341714049" watchObservedRunningTime="2026-10-30 22:27:26.969856962 +0000 UTC m=+2704.342483710" Oct 30 22:27:27.044324 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:27.044286 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:27.045979 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:27.045935 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:27:27.046100 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:27.046011 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:27:28.044800 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:28.044762 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:27:28.045253 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:28.044822 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:27:29.044671 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:29.044639 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:27:29.044868 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:29.044693 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:27:30.044631 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:30.044589 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:27:30.044806 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:30.044647 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:27:31.044324 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:31.044287 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:27:31.044700 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:31.044341 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:27:32.043867 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:32.043830 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:27:32.044065 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:32.043888 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:27:33.044410 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:33.044373 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:27:33.044824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:33.044432 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:27:35.086142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:35.086082 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:27:35.086643 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:35.086196 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:35.215044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:35.215013 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:35.215454 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:35.215430 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:35.216082 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:35.216064 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:27:39.876878 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:27:39.876844 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:27:45.179032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.178989 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc"] Oct 30 22:27:45.184045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.184022 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.329226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.328746 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-istio-token\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.329226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.328789 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d91468a-3e76-485e-8cac-123a4e839b86-istiod-ca-cert\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.329226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.328813 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-envoy\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.329226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.328870 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-data\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.329226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.328895 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-certs\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.329226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.328917 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9jj\" (UniqueName: \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-kube-api-access-bd9jj\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.329226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.329003 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-credential-socket\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.329226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.329022 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-socket\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.329226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.329069 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d91468a-3e76-485e-8cac-123a4e839b86-istio-podinfo\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.334285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.334240 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b563fd-openshift-default-dockercfg-5jqwc\"" Oct 30 22:27:45.339506 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.335796 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc"] Oct 30 22:27:45.429895 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.429803 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-data\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.429895 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.429854 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-certs\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.429895 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.429873 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9jj\" (UniqueName: \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-kube-api-access-bd9jj\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430201 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.429929 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-credential-socket\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430201 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.429963 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-socket\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430201 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.430008 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d91468a-3e76-485e-8cac-123a4e839b86-istio-podinfo\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430201 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.430095 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-istio-token\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430201 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.430125 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d91468a-3e76-485e-8cac-123a4e839b86-istiod-ca-cert\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430201 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.430160 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-envoy\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.430224 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-data\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.430276 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-certs\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.430342 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-credential-socket\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430629 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.430609 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-socket\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.430870 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.430853 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d91468a-3e76-485e-8cac-123a4e839b86-istiod-ca-cert\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.432357 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.432336 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-envoy\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.432616 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.432600 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d91468a-3e76-485e-8cac-123a4e839b86-istio-podinfo\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.488213 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.488180 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9jj\" (UniqueName: \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-kube-api-access-bd9jj\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.488396 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.488377 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-istio-token\") pod \"e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.522222 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.521168 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:45.972554 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:27:45.972522 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d91468a_3e76_485e_8cac_123a4e839b86.slice/crio-7ff36be87ae6fe4300166db5af769e7e9ed06b51b584a49f386ee07797aa7437 WatchSource:0}: Error finding container 7ff36be87ae6fe4300166db5af769e7e9ed06b51b584a49f386ee07797aa7437: Status 404 returned error can't find the container with id 7ff36be87ae6fe4300166db5af769e7e9ed06b51b584a49f386ee07797aa7437 Oct 30 22:27:45.972696 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.972675 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc"] Oct 30 22:27:45.975029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.974996 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:45.975111 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.975060 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:45.975111 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:45.975093 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:46.054900 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:46.054872 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" event={"ID":"1d91468a-3e76-485e-8cac-123a4e839b86","Type":"ContainerStarted","Data":"21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274"} Oct 30 22:27:46.055019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:46.054920 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" event={"ID":"1d91468a-3e76-485e-8cac-123a4e839b86","Type":"ContainerStarted","Data":"7ff36be87ae6fe4300166db5af769e7e9ed06b51b584a49f386ee07797aa7437"} Oct 30 22:27:46.182300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:46.182242 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" podStartSLOduration=1.182227565 podStartE2EDuration="1.182227565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:46.178346116 +0000 UTC m=+2723.550972863" watchObservedRunningTime="2026-10-30 22:27:46.182227565 +0000 UTC m=+2723.554854311" Oct 30 22:27:46.522054 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:46.522022 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:47.527809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:47.527778 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:48.066321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:48.066287 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:27:48.067241 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:27:48.067222 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:28:09.879704 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:28:09.879627 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:28:19.724846 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:19.724805 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:28:23.832887 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:23.832845 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v"] Oct 30 22:28:23.833413 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:23.833108 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" containerID="cri-o://c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103" gracePeriod=30 Oct 30 22:28:29.095101 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.095075 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:28:29.147226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147136 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-kube-api-access-hrd8l\" (UniqueName: \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-kube-api-access-hrd8l\") pod \"5d209f91-4095-47a2-866d-d58f89e198a2\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " Oct 30 22:28:29.147226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147185 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-credential-socket\") pod \"5d209f91-4095-47a2-866d-d58f89e198a2\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " Oct 30 22:28:29.147440 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147238 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-envoy\") pod \"5d209f91-4095-47a2-866d-d58f89e198a2\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " Oct 30 22:28:29.147440 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147318 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d209f91-4095-47a2-866d-d58f89e198a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d209f91-4095-47a2-866d-d58f89e198a2-istiod-ca-cert\") pod \"5d209f91-4095-47a2-866d-d58f89e198a2\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " Oct 30 22:28:29.147440 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147388 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-socket\") pod \"5d209f91-4095-47a2-866d-d58f89e198a2\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " Oct 30 22:28:29.147607 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147499 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d209f91-4095-47a2-866d-d58f89e198a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d209f91-4095-47a2-866d-d58f89e198a2-istio-podinfo\") pod \"5d209f91-4095-47a2-866d-d58f89e198a2\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " Oct 30 22:28:29.147607 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147553 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-certs\") pod \"5d209f91-4095-47a2-866d-d58f89e198a2\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " Oct 30 22:28:29.147607 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147579 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-credential-socket" pod "5d209f91-4095-47a2-866d-d58f89e198a2" (UID: "5d209f91-4095-47a2-866d-d58f89e198a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:29.147770 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147613 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-data\") pod \"5d209f91-4095-47a2-866d-d58f89e198a2\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " Oct 30 22:28:29.147770 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147673 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-socket" pod "5d209f91-4095-47a2-866d-d58f89e198a2" (UID: "5d209f91-4095-47a2-866d-d58f89e198a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:29.147770 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147737 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-istio-token\" (UniqueName: \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-istio-token\") pod \"5d209f91-4095-47a2-866d-d58f89e198a2\" (UID: \"5d209f91-4095-47a2-866d-d58f89e198a2\") " Oct 30 22:28:29.147933 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147856 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-certs" pod "5d209f91-4095-47a2-866d-d58f89e198a2" (UID: "5d209f91-4095-47a2-866d-d58f89e198a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:29.147933 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.147868 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-data" pod "5d209f91-4095-47a2-866d-d58f89e198a2" (UID: "5d209f91-4095-47a2-866d-d58f89e198a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:29.148088 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.148060 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d209f91-4095-47a2-866d-d58f89e198a2-istiod-ca-cert" pod "5d209f91-4095-47a2-866d-d58f89e198a2" (UID: "5d209f91-4095-47a2-866d-d58f89e198a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:29.148352 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.148327 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.148453 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.148357 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d209f91-4095-47a2-866d-d58f89e198a2-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.148453 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.148373 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.148453 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.148388 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.148453 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.148403 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.149566 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.149531 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-envoy" pod "5d209f91-4095-47a2-866d-d58f89e198a2" (UID: "5d209f91-4095-47a2-866d-d58f89e198a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:29.149878 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.149847 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d209f91-4095-47a2-866d-d58f89e198a2-istio-podinfo" pod "5d209f91-4095-47a2-866d-d58f89e198a2" (UID: "5d209f91-4095-47a2-866d-d58f89e198a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:29.150219 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.150196 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-istio-token" pod "5d209f91-4095-47a2-866d-d58f89e198a2" (UID: "5d209f91-4095-47a2-866d-d58f89e198a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:29.150312 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.150271 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-kube-api-access-hrd8l" pod "5d209f91-4095-47a2-866d-d58f89e198a2" (UID: "5d209f91-4095-47a2-866d-d58f89e198a2"). InnerVolumeSpecName "kube-api-access-hrd8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:29.249599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.249568 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrd8l\" (UniqueName: \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-kube-api-access-hrd8l\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.249599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.249597 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d209f91-4095-47a2-866d-d58f89e198a2-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.249599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.249606 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d209f91-4095-47a2-866d-d58f89e198a2-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.249831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.249616 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d209f91-4095-47a2-866d-d58f89e198a2-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.450365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.450264 4859 generic.go:356] "Generic (PLEG): container finished" podID="5d209f91-4095-47a2-866d-d58f89e198a2" containerID="c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103" exitCode=0 Oct 30 22:28:29.450365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.450332 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" Oct 30 22:28:29.450365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.450350 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" event={"ID":"5d209f91-4095-47a2-866d-d58f89e198a2","Type":"ContainerDied","Data":"c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103"} Oct 30 22:28:29.450653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.450425 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v" event={"ID":"5d209f91-4095-47a2-866d-d58f89e198a2","Type":"ContainerDied","Data":"b561d982682c9298d2516a9449e5185ad9ec45cc4adc71a7d49f1510dc26cd79"} Oct 30 22:28:29.450653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.450457 4859 scope.go:122] "RemoveContainer" containerID="c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103" Oct 30 22:28:29.462047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.462028 4859 scope.go:122] "RemoveContainer" containerID="bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3" Oct 30 22:28:29.471998 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.471967 4859 scope.go:122] "RemoveContainer" containerID="c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103" Oct 30 22:28:29.472356 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:28:29.472329 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103\": container with ID starting with c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103 not found: ID does not exist" containerID="c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103" Oct 30 22:28:29.472429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.472369 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103"} err="failed to get container status \"c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103\": rpc error: code = NotFound desc = could not find container \"c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103\": container with ID starting with c1d15d9701db2d59cd7e7dad8894840030aac9cd319d3a9415c1c3664b43e103 not found: ID does not exist" Oct 30 22:28:29.472429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.472394 4859 scope.go:122] "RemoveContainer" containerID="bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3" Oct 30 22:28:29.472723 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:28:29.472703 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3\": container with ID starting with bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3 not found: ID does not exist" containerID="bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3" Oct 30 22:28:29.472788 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.472728 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3"} err="failed to get container status \"bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3\": rpc error: code = NotFound desc = could not find container \"bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3\": container with ID starting with bd566b81ebd6d29984c79d4657824445ce744e80881853ce8fc5f369286cd5f3 not found: ID does not exist" Oct 30 22:28:29.617512 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.617471 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v"] Oct 30 22:28:29.621661 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:29.621632 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-cce50a-openshift-default-5666c7db6f-8qv2v"] Oct 30 22:28:30.742314 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:30.742278 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" path="/var/lib/kubelet/pods/5d209f91-4095-47a2-866d-d58f89e198a2/volumes" Oct 30 22:28:36.459619 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.459580 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n"] Oct 30 22:28:36.460088 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.459770 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" podUID="9cdd38ed-b9be-4835-9b42-1e508a764574" containerName="maas-api" containerID="cri-o://dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f" gracePeriod=30 Oct 30 22:28:36.568130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.568080 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" podUID="9cdd38ed-b9be-4835-9b42-1e508a764574" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.140:8443/health\": dial tcp 10.132.0.140:8443: connect: connection refused" Oct 30 22:28:36.782093 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.782024 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:28:36.886008 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.879267 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-metrics-certs\") pod \"9cdd38ed-b9be-4835-9b42-1e508a764574\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " Oct 30 22:28:36.886305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.886271 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-maas-api-tls\") pod \"9cdd38ed-b9be-4835-9b42-1e508a764574\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " Oct 30 22:28:36.886443 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.886388 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cdd38ed-b9be-4835-9b42-1e508a764574-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdd38ed-b9be-4835-9b42-1e508a764574-openshift-service-ca\") pod \"9cdd38ed-b9be-4835-9b42-1e508a764574\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " Oct 30 22:28:36.886443 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.886429 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdd38ed-b9be-4835-9b42-1e508a764574-kube-api-access-sbw9t\" (UniqueName: \"kubernetes.io/projected/9cdd38ed-b9be-4835-9b42-1e508a764574-kube-api-access-sbw9t\") pod \"9cdd38ed-b9be-4835-9b42-1e508a764574\" (UID: \"9cdd38ed-b9be-4835-9b42-1e508a764574\") " Oct 30 22:28:36.900145 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.899841 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdd38ed-b9be-4835-9b42-1e508a764574-openshift-service-ca" pod "9cdd38ed-b9be-4835-9b42-1e508a764574" (UID: "9cdd38ed-b9be-4835-9b42-1e508a764574"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:36.914231 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.912789 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-maas-api-tls" pod "9cdd38ed-b9be-4835-9b42-1e508a764574" (UID: "9cdd38ed-b9be-4835-9b42-1e508a764574"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:36.914231 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.912882 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-metrics-certs" pod "9cdd38ed-b9be-4835-9b42-1e508a764574" (UID: "9cdd38ed-b9be-4835-9b42-1e508a764574"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:36.974235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:36.974179 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdd38ed-b9be-4835-9b42-1e508a764574-kube-api-access-sbw9t" pod "9cdd38ed-b9be-4835-9b42-1e508a764574" (UID: "9cdd38ed-b9be-4835-9b42-1e508a764574"). InnerVolumeSpecName "kube-api-access-sbw9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:37.010966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.007312 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.010966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.007350 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cdd38ed-b9be-4835-9b42-1e508a764574-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.010966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.007393 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdd38ed-b9be-4835-9b42-1e508a764574-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.010966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.007410 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbw9t\" (UniqueName: \"kubernetes.io/projected/9cdd38ed-b9be-4835-9b42-1e508a764574-kube-api-access-sbw9t\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.525013 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.520457 4859 generic.go:356] "Generic (PLEG): container finished" podID="9cdd38ed-b9be-4835-9b42-1e508a764574" containerID="dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f" exitCode=0 Oct 30 22:28:37.525013 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.520767 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" event={"ID":"9cdd38ed-b9be-4835-9b42-1e508a764574","Type":"ContainerDied","Data":"dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f"} Oct 30 22:28:37.525013 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.520860 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" event={"ID":"9cdd38ed-b9be-4835-9b42-1e508a764574","Type":"ContainerDied","Data":"c346138e7d9e9325d778cd960b483b31e57b68ad56ca79fb75f010ff9ecbe178"} Oct 30 22:28:37.525013 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.520892 4859 scope.go:122] "RemoveContainer" containerID="dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f" Oct 30 22:28:37.525013 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.521072 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n" Oct 30 22:28:37.592967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.591131 4859 scope.go:122] "RemoveContainer" containerID="dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f" Oct 30 22:28:37.596961 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:28:37.594100 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f\": container with ID starting with dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f not found: ID does not exist" containerID="dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f" Oct 30 22:28:37.596961 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.594148 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f"} err="failed to get container status \"dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f\": rpc error: code = NotFound desc = could not find container \"dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f\": container with ID starting with dfd3cb46b8e3e456afe9b3a9f36f561ed882f72be967011a101a81c059cd903f not found: ID does not exist" Oct 30 22:28:37.705575 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.705530 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n"] Oct 30 22:28:37.754859 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:37.754785 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5b1e39-df8658ddc-zjm4n"] Oct 30 22:28:38.741228 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:38.741198 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cdd38ed-b9be-4835-9b42-1e508a764574" path="/var/lib/kubelet/pods/9cdd38ed-b9be-4835-9b42-1e508a764574/volumes" Oct 30 22:28:39.898375 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:28:39.898340 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:28:43.095342 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.095307 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2"] Oct 30 22:28:43.096200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.096176 4859 cpu_manager.go:414] "containerMap: removing container" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" Oct 30 22:28:43.096385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.096364 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" Oct 30 22:28:43.096521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.096508 4859 cpu_manager.go:414] "containerMap: removing container" podUID="9cdd38ed-b9be-4835-9b42-1e508a764574" containerName="maas-api" Oct 30 22:28:43.096611 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.096598 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdd38ed-b9be-4835-9b42-1e508a764574" containerName="maas-api" Oct 30 22:28:43.096741 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.096708 4859 cpu_manager.go:414] "containerMap: removing container" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" Oct 30 22:28:43.096741 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.096725 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" Oct 30 22:28:43.096926 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.096887 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" Oct 30 22:28:43.096926 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.096915 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdd38ed-b9be-4835-9b42-1e508a764574" containerName="maas-api" Oct 30 22:28:43.101694 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.101674 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.154594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.154562 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b563fd-serving-cert\"" Oct 30 22:28:43.207364 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.207333 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2"] Oct 30 22:28:43.208578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.208554 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.208648 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.208602 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggdl\" (UniqueName: \"kubernetes.io/projected/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-kube-api-access-6ggdl\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.208778 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.208763 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.208818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.208790 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-metrics-certs\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.310031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.309991 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.310226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.310073 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggdl\" (UniqueName: \"kubernetes.io/projected/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-kube-api-access-6ggdl\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.310226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.310201 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.310339 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.310234 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-metrics-certs\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.310513 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:28:43.310479 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-serving-cert: secret "maas-api-e2e-praxis-b-b563fd-serving-cert" not found Oct 30 22:28:43.310607 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:28:43.310550 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-maas-api-tls podName:1f54f71b-4cc8-4335-a50a-a34cb07e7c52 nodeName:}" failed. No retries permitted until 2026-10-30 22:28:43.810528481 +0000 UTC m=+2781.183155219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-maas-api-tls") pod "maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" (UID: "1f54f71b-4cc8-4335-a50a-a34cb07e7c52") : secret "maas-api-e2e-praxis-b-b563fd-serving-cert" not found Oct 30 22:28:43.310887 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.310856 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.313169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.313148 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-metrics-certs\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.394014 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.393148 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggdl\" (UniqueName: \"kubernetes.io/projected/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-kube-api-access-6ggdl\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.819278 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.819188 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:43.821772 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:43.821740 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:44.015725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:44.015697 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:44.402307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:44.402272 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2"] Oct 30 22:28:44.404689 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:28:44.404657 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f54f71b_4cc8_4335_a50a_a34cb07e7c52.slice/crio-73d230fccf5d8f7580e5bc3c8e33652b3027f7d11266c7606fe62aff1e1e82f6 WatchSource:0}: Error finding container 73d230fccf5d8f7580e5bc3c8e33652b3027f7d11266c7606fe62aff1e1e82f6: Status 404 returned error can't find the container with id 73d230fccf5d8f7580e5bc3c8e33652b3027f7d11266c7606fe62aff1e1e82f6 Oct 30 22:28:44.578286 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:44.578248 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" event={"ID":"1f54f71b-4cc8-4335-a50a-a34cb07e7c52","Type":"ContainerStarted","Data":"73d230fccf5d8f7580e5bc3c8e33652b3027f7d11266c7606fe62aff1e1e82f6"} Oct 30 22:28:45.587239 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:45.587203 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" event={"ID":"1f54f71b-4cc8-4335-a50a-a34cb07e7c52","Type":"ContainerStarted","Data":"3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37"} Oct 30 22:28:45.588405 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:45.587346 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:45.704580 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:45.704535 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" podStartSLOduration=1.82129274 podStartE2EDuration="2.704521646s" totalImagesPullingTime="883.228906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:45.702758119 +0000 UTC m=+2783.075384890" watchObservedRunningTime="2026-10-30 22:28:45.704521646 +0000 UTC m=+2783.077148392" Oct 30 22:28:46.596457 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:46.596427 4859 generic.go:356] "Generic (PLEG): container finished" podID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerID="f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe" exitCode=0 Oct 30 22:28:46.596907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:46.596503 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" event={"ID":"e010b301-bf27-4059-ad1c-0e9e94d05684","Type":"ContainerDied","Data":"f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe"} Oct 30 22:28:46.597209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:46.597191 4859 scope.go:122] "RemoveContainer" containerID="f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe" Oct 30 22:28:46.598466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:46.598434 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:46.598566 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:46.598519 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:46.598628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:46.598570 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:47.605743 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:47.605706 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" event={"ID":"e010b301-bf27-4059-ad1c-0e9e94d05684","Type":"ContainerStarted","Data":"37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923"} Oct 30 22:28:48.581616 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:48.581587 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:28:48.581956 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:48.581921 4859 patch_prober.go:29] interesting pod/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:28:48.582130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:48.582104 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:28:49.580791 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:49.580346 4859 patch_prober.go:29] interesting pod/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:28:49.580791 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:49.580416 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:28:49.800637 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:49.800596 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:28:50.581967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:50.580348 4859 patch_prober.go:29] interesting pod/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:28:50.581967 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:50.580409 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:28:51.583968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:51.583406 4859 patch_prober.go:29] interesting pod/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:28:51.583968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:51.583473 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:28:51.602491 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:51.602459 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:28:52.585125 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:52.585095 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:28:52.586344 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:52.586320 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:28:56.983783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:28:56.983750 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:29:06.825274 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:06.825234 4859 generic.go:356] "Generic (PLEG): container finished" podID="1d91468a-3e76-485e-8cac-123a4e839b86" containerID="21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274" exitCode=0 Oct 30 22:29:06.825785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:06.825379 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" event={"ID":"1d91468a-3e76-485e-8cac-123a4e839b86","Type":"ContainerDied","Data":"21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274"} Oct 30 22:29:06.825785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:06.825679 4859 scope.go:122] "RemoveContainer" containerID="21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274" Oct 30 22:29:06.827323 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:06.827289 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:29:06.827431 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:06.827361 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:29:06.827431 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:06.827392 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:29:07.834203 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:07.834159 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" event={"ID":"1d91468a-3e76-485e-8cac-123a4e839b86","Type":"ContainerStarted","Data":"842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe"} Oct 30 22:29:08.522592 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:08.522557 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:29:08.522781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:08.522716 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:29:08.522855 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:08.522771 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 22:29:09.522441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:09.522408 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:29:09.522903 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:09.522469 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 22:29:09.896537 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:29:09.896485 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:29:10.522092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:10.522059 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:29:10.522321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:10.522120 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 22:29:11.522588 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:11.522551 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:29:11.522994 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:11.522608 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 22:29:12.522054 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:12.522017 4859 patch_prober.go:29] interesting pod/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:29:12.522238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:12.522071 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 22:29:13.525933 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:13.525903 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:29:13.526407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:13.526395 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:29:13.527095 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:13.527071 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:29:39.878384 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:29:39.878301 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:29:50.442930 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:29:50.442897 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:30:00.170894 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.170857 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx"] Oct 30 22:30:00.171544 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.171527 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d209f91-4095-47a2-866d-d58f89e198a2" containerName="istio-proxy" Oct 30 22:30:00.174314 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.174298 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" Oct 30 22:30:00.175592 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.175568 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d"] Oct 30 22:30:00.179835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.179819 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" Oct 30 22:30:00.195916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.195892 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9"] Oct 30 22:30:00.199528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.199511 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" Oct 30 22:30:00.218262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.218239 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb"] Oct 30 22:30:00.222775 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.222755 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8"] Oct 30 22:30:00.222875 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.222866 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" Oct 30 22:30:00.226480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.226464 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" Oct 30 22:30:00.242326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.242309 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\"" Oct 30 22:30:00.274222 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.274196 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d"] Oct 30 22:30:00.278651 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.278630 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx"] Oct 30 22:30:00.299465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.298479 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bea4c7-cd12-4ed9-ae40-5040e655e562-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb\" (UID: \"48bea4c7-cd12-4ed9-ae40-5040e655e562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" Oct 30 22:30:00.299465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.298563 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2xz\" (UniqueName: \"kubernetes.io/projected/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-kube-api-access-ml2xz\") pod \"maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9\" (UID: \"e5f5f4fb-7141-43d0-be2c-3217a7e45f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" Oct 30 22:30:00.299465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.298602 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea61988-fb62-4427-aea8-5e7b5360ada5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d\" (UID: \"4ea61988-fb62-4427-aea8-5e7b5360ada5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" Oct 30 22:30:00.299465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.298642 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwkjt\" (UniqueName: \"kubernetes.io/projected/4ea61988-fb62-4427-aea8-5e7b5360ada5-kube-api-access-lwkjt\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d\" (UID: \"4ea61988-fb62-4427-aea8-5e7b5360ada5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" Oct 30 22:30:00.299465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.298706 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50f5739-92bc-4948-bc40-17e904811f40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx\" (UID: \"c50f5739-92bc-4948-bc40-17e904811f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" Oct 30 22:30:00.299465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.298744 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdt8b\" (UniqueName: \"kubernetes.io/projected/48bea4c7-cd12-4ed9-ae40-5040e655e562-kube-api-access-sdt8b\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb\" (UID: \"48bea4c7-cd12-4ed9-ae40-5040e655e562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" Oct 30 22:30:00.299465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.298816 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfsdn\" (UniqueName: \"kubernetes.io/projected/c50f5739-92bc-4948-bc40-17e904811f40-kube-api-access-mfsdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx\" (UID: \"c50f5739-92bc-4948-bc40-17e904811f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" Oct 30 22:30:00.299465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.298852 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9\" (UID: \"e5f5f4fb-7141-43d0-be2c-3217a7e45f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" Oct 30 22:30:00.299465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.298895 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfbd\" (UniqueName: \"kubernetes.io/projected/13f89a58-0eaf-4da2-b9f4-684618c3187f-kube-api-access-tzfbd\") pod \"maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8\" (UID: \"13f89a58-0eaf-4da2-b9f4-684618c3187f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" Oct 30 22:30:00.299465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.298966 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f89a58-0eaf-4da2-b9f4-684618c3187f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8\" (UID: \"13f89a58-0eaf-4da2-b9f4-684618c3187f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" Oct 30 22:30:00.323094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.320255 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9"] Oct 30 22:30:00.325309 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.325283 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb"] Oct 30 22:30:00.373623 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.373585 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8"] Oct 30 22:30:00.400337 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.400309 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50f5739-92bc-4948-bc40-17e904811f40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx\" (UID: \"c50f5739-92bc-4948-bc40-17e904811f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" Oct 30 22:30:00.400504 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.400371 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdt8b\" (UniqueName: \"kubernetes.io/projected/48bea4c7-cd12-4ed9-ae40-5040e655e562-kube-api-access-sdt8b\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb\" (UID: \"48bea4c7-cd12-4ed9-ae40-5040e655e562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" Oct 30 22:30:00.400504 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.400460 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfsdn\" (UniqueName: \"kubernetes.io/projected/c50f5739-92bc-4948-bc40-17e904811f40-kube-api-access-mfsdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx\" (UID: \"c50f5739-92bc-4948-bc40-17e904811f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" Oct 30 22:30:00.400504 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.400490 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9\" (UID: \"e5f5f4fb-7141-43d0-be2c-3217a7e45f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" Oct 30 22:30:00.400793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.400529 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfbd\" (UniqueName: \"kubernetes.io/projected/13f89a58-0eaf-4da2-b9f4-684618c3187f-kube-api-access-tzfbd\") pod \"maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8\" (UID: \"13f89a58-0eaf-4da2-b9f4-684618c3187f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" Oct 30 22:30:00.400793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.400580 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f89a58-0eaf-4da2-b9f4-684618c3187f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8\" (UID: \"13f89a58-0eaf-4da2-b9f4-684618c3187f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" Oct 30 22:30:00.400793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.400640 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bea4c7-cd12-4ed9-ae40-5040e655e562-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb\" (UID: \"48bea4c7-cd12-4ed9-ae40-5040e655e562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" Oct 30 22:30:00.400793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.400722 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2xz\" (UniqueName: \"kubernetes.io/projected/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-kube-api-access-ml2xz\") pod \"maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9\" (UID: \"e5f5f4fb-7141-43d0-be2c-3217a7e45f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" Oct 30 22:30:00.400793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.400755 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea61988-fb62-4427-aea8-5e7b5360ada5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d\" (UID: \"4ea61988-fb62-4427-aea8-5e7b5360ada5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" Oct 30 22:30:00.401228 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.400791 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwkjt\" (UniqueName: \"kubernetes.io/projected/4ea61988-fb62-4427-aea8-5e7b5360ada5-kube-api-access-lwkjt\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d\" (UID: \"4ea61988-fb62-4427-aea8-5e7b5360ada5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" Oct 30 22:30:00.401407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.401370 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9\" (UID: \"e5f5f4fb-7141-43d0-be2c-3217a7e45f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" Oct 30 22:30:00.401630 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.401610 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50f5739-92bc-4948-bc40-17e904811f40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx\" (UID: \"c50f5739-92bc-4948-bc40-17e904811f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" Oct 30 22:30:00.401855 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.401832 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f89a58-0eaf-4da2-b9f4-684618c3187f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8\" (UID: \"13f89a58-0eaf-4da2-b9f4-684618c3187f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" Oct 30 22:30:00.402488 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.402463 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bea4c7-cd12-4ed9-ae40-5040e655e562-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb\" (UID: \"48bea4c7-cd12-4ed9-ae40-5040e655e562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" Oct 30 22:30:00.402575 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.402505 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea61988-fb62-4427-aea8-5e7b5360ada5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d\" (UID: \"4ea61988-fb62-4427-aea8-5e7b5360ada5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" Oct 30 22:30:00.458335 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.457870 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwkjt\" (UniqueName: \"kubernetes.io/projected/4ea61988-fb62-4427-aea8-5e7b5360ada5-kube-api-access-lwkjt\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d\" (UID: \"4ea61988-fb62-4427-aea8-5e7b5360ada5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" Oct 30 22:30:00.458335 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.458281 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfsdn\" (UniqueName: \"kubernetes.io/projected/c50f5739-92bc-4948-bc40-17e904811f40-kube-api-access-mfsdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx\" (UID: \"c50f5739-92bc-4948-bc40-17e904811f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" Oct 30 22:30:00.458684 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.458658 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfbd\" (UniqueName: \"kubernetes.io/projected/13f89a58-0eaf-4da2-b9f4-684618c3187f-kube-api-access-tzfbd\") pod \"maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8\" (UID: \"13f89a58-0eaf-4da2-b9f4-684618c3187f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" Oct 30 22:30:00.459107 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.459077 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2xz\" (UniqueName: \"kubernetes.io/projected/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-kube-api-access-ml2xz\") pod \"maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9\" (UID: \"e5f5f4fb-7141-43d0-be2c-3217a7e45f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" Oct 30 22:30:00.460121 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.460099 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdt8b\" (UniqueName: \"kubernetes.io/projected/48bea4c7-cd12-4ed9-ae40-5040e655e562-kube-api-access-sdt8b\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb\" (UID: \"48bea4c7-cd12-4ed9-ae40-5040e655e562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" Oct 30 22:30:00.488328 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.488301 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" Oct 30 22:30:00.495979 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.495930 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" Oct 30 22:30:00.521359 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.514258 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" Oct 30 22:30:00.539966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.538145 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" Oct 30 22:30:00.547360 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.547159 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" Oct 30 22:30:00.896727 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.896695 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx"] Oct 30 22:30:00.897166 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:30:00.897137 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50f5739_92bc_4948_bc40_17e904811f40.slice/crio-8da8ba621c115aa636f2a7137f4635d0afdc6239fb4a69e6b998ef1dfa4acebd WatchSource:0}: Error finding container 8da8ba621c115aa636f2a7137f4635d0afdc6239fb4a69e6b998ef1dfa4acebd: Status 404 returned error can't find the container with id 8da8ba621c115aa636f2a7137f4635d0afdc6239fb4a69e6b998ef1dfa4acebd Oct 30 22:30:00.947266 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.947243 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d"] Oct 30 22:30:00.947815 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:30:00.947795 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea61988_fb62_4427_aea8_5e7b5360ada5.slice/crio-56c5efc9829fa1a612e50ef40308184d5abdaf3674e355fc712880ec995b341d WatchSource:0}: Error finding container 56c5efc9829fa1a612e50ef40308184d5abdaf3674e355fc712880ec995b341d: Status 404 returned error can't find the container with id 56c5efc9829fa1a612e50ef40308184d5abdaf3674e355fc712880ec995b341d Oct 30 22:30:00.950013 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:30:00.949990 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f5f4fb_7141_43d0_be2c_3217a7e45f84.slice/crio-9c4f2cc98f64f57071d8d41c23ffccf3ecfee1f30dfe1f383fa22aad003d090b WatchSource:0}: Error finding container 9c4f2cc98f64f57071d8d41c23ffccf3ecfee1f30dfe1f383fa22aad003d090b: Status 404 returned error can't find the container with id 9c4f2cc98f64f57071d8d41c23ffccf3ecfee1f30dfe1f383fa22aad003d090b Oct 30 22:30:00.950765 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:00.950707 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9"] Oct 30 22:30:01.004659 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:30:01.000273 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f89a58_0eaf_4da2_b9f4_684618c3187f.slice/crio-57bc4bbba4d897a1b276e1b48231647101c390395b85b9c63472084cef15d8a9 WatchSource:0}: Error finding container 57bc4bbba4d897a1b276e1b48231647101c390395b85b9c63472084cef15d8a9: Status 404 returned error can't find the container with id 57bc4bbba4d897a1b276e1b48231647101c390395b85b9c63472084cef15d8a9 Oct 30 22:30:01.004659 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:01.000456 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8"] Oct 30 22:30:01.006692 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:01.005357 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb"] Oct 30 22:30:01.013261 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:30:01.010740 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bea4c7_cd12_4ed9_ae40_5040e655e562.slice/crio-5b848771d2fe34f4eb9c4867b007aa558d51ff67ca3156ae30015b3971dba2fc WatchSource:0}: Error finding container 5b848771d2fe34f4eb9c4867b007aa558d51ff67ca3156ae30015b3971dba2fc: Status 404 returned error can't find the container with id 5b848771d2fe34f4eb9c4867b007aa558d51ff67ca3156ae30015b3971dba2fc Oct 30 22:30:01.312781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:01.312743 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" event={"ID":"e5f5f4fb-7141-43d0-be2c-3217a7e45f84","Type":"ContainerStarted","Data":"9c4f2cc98f64f57071d8d41c23ffccf3ecfee1f30dfe1f383fa22aad003d090b"} Oct 30 22:30:01.314130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:01.314099 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" event={"ID":"4ea61988-fb62-4427-aea8-5e7b5360ada5","Type":"ContainerStarted","Data":"56c5efc9829fa1a612e50ef40308184d5abdaf3674e355fc712880ec995b341d"} Oct 30 22:30:01.315464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:01.315434 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" event={"ID":"48bea4c7-cd12-4ed9-ae40-5040e655e562","Type":"ContainerStarted","Data":"5b848771d2fe34f4eb9c4867b007aa558d51ff67ca3156ae30015b3971dba2fc"} Oct 30 22:30:01.316564 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:01.316542 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" event={"ID":"13f89a58-0eaf-4da2-b9f4-684618c3187f","Type":"ContainerStarted","Data":"57bc4bbba4d897a1b276e1b48231647101c390395b85b9c63472084cef15d8a9"} Oct 30 22:30:01.317709 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:01.317683 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" event={"ID":"c50f5739-92bc-4948-bc40-17e904811f40","Type":"ContainerStarted","Data":"8da8ba621c115aa636f2a7137f4635d0afdc6239fb4a69e6b998ef1dfa4acebd"} Oct 30 22:30:02.327301 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.327231 4859 generic.go:356] "Generic (PLEG): container finished" podID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerID="79b9198d3ff32e52b5688d0b4b4985ce86301716e7461af36375cd426fc4d4da" exitCode=60 Oct 30 22:30:02.327668 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.327319 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" event={"ID":"e5f5f4fb-7141-43d0-be2c-3217a7e45f84","Type":"ContainerDied","Data":"79b9198d3ff32e52b5688d0b4b4985ce86301716e7461af36375cd426fc4d4da"} Oct 30 22:30:02.327668 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.327518 4859 scope.go:122] "RemoveContainer" containerID="79b9198d3ff32e52b5688d0b4b4985ce86301716e7461af36375cd426fc4d4da" Oct 30 22:30:02.328999 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.328978 4859 generic.go:356] "Generic (PLEG): container finished" podID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerID="225d60cf4d81e4e5f146c89eaf8bba63f03a84217f36feeb127ff75bc076814d" exitCode=60 Oct 30 22:30:02.329145 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.329057 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" event={"ID":"4ea61988-fb62-4427-aea8-5e7b5360ada5","Type":"ContainerDied","Data":"225d60cf4d81e4e5f146c89eaf8bba63f03a84217f36feeb127ff75bc076814d"} Oct 30 22:30:02.329242 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.329226 4859 scope.go:122] "RemoveContainer" containerID="225d60cf4d81e4e5f146c89eaf8bba63f03a84217f36feeb127ff75bc076814d" Oct 30 22:30:02.330689 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.330669 4859 generic.go:356] "Generic (PLEG): container finished" podID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerID="381dd823d77468b677c12577fc547cde00aa2c8ada12d2c6da2558e08f0a345b" exitCode=60 Oct 30 22:30:02.330772 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.330748 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" event={"ID":"48bea4c7-cd12-4ed9-ae40-5040e655e562","Type":"ContainerDied","Data":"381dd823d77468b677c12577fc547cde00aa2c8ada12d2c6da2558e08f0a345b"} Oct 30 22:30:02.330917 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.330900 4859 scope.go:122] "RemoveContainer" containerID="381dd823d77468b677c12577fc547cde00aa2c8ada12d2c6da2558e08f0a345b" Oct 30 22:30:02.332347 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.332321 4859 generic.go:356] "Generic (PLEG): container finished" podID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerID="b2cd9fb04bab8b1d0f2d84da841f9339043cd857ec5c48f4d2a0a85f82a169f1" exitCode=60 Oct 30 22:30:02.332417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.332399 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" event={"ID":"13f89a58-0eaf-4da2-b9f4-684618c3187f","Type":"ContainerDied","Data":"b2cd9fb04bab8b1d0f2d84da841f9339043cd857ec5c48f4d2a0a85f82a169f1"} Oct 30 22:30:02.332576 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.332561 4859 scope.go:122] "RemoveContainer" containerID="b2cd9fb04bab8b1d0f2d84da841f9339043cd857ec5c48f4d2a0a85f82a169f1" Oct 30 22:30:02.333878 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.333861 4859 generic.go:356] "Generic (PLEG): container finished" podID="c50f5739-92bc-4948-bc40-17e904811f40" containerID="336048c1d21f81ce25a797dd949b6cb09c62005472dc2b9b945b15b2e88bad5f" exitCode=60 Oct 30 22:30:02.333965 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.333884 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" event={"ID":"c50f5739-92bc-4948-bc40-17e904811f40","Type":"ContainerDied","Data":"336048c1d21f81ce25a797dd949b6cb09c62005472dc2b9b945b15b2e88bad5f"} Oct 30 22:30:02.334143 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:02.334127 4859 scope.go:122] "RemoveContainer" containerID="336048c1d21f81ce25a797dd949b6cb09c62005472dc2b9b945b15b2e88bad5f" Oct 30 22:30:03.343129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:03.343098 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" event={"ID":"48bea4c7-cd12-4ed9-ae40-5040e655e562","Type":"ContainerStarted","Data":"23122602406b47c0b60eda601d045ca684e0070b1b255b76768235600f6c6fc4"} Oct 30 22:30:03.344974 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:03.344934 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" event={"ID":"13f89a58-0eaf-4da2-b9f4-684618c3187f","Type":"ContainerStarted","Data":"d56b41c243e2c4a84d55fb2268da45ff7f67f5b8fa54e5c142cdb77aa78a87db"} Oct 30 22:30:03.346813 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:03.346788 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" event={"ID":"c50f5739-92bc-4948-bc40-17e904811f40","Type":"ContainerStarted","Data":"495ab785c240ba74c289d6fd4ec083c66cb35cbf589647c04d023c057569f3e3"} Oct 30 22:30:03.348648 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:03.348624 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" event={"ID":"e5f5f4fb-7141-43d0-be2c-3217a7e45f84","Type":"ContainerStarted","Data":"6f3ca6ee307863462a879d533b661109907038d3709af2e2c4819ac4326a6a87"} Oct 30 22:30:03.350366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:03.350345 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" event={"ID":"4ea61988-fb62-4427-aea8-5e7b5360ada5","Type":"ContainerStarted","Data":"f5e5a61bf521fc1fc8ad6f1cac97787db82a35a3c3cc167205b191e7e7f34151"} Oct 30 22:30:03.464887 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:03.464837 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" podStartSLOduration=3.46482483 podStartE2EDuration="3.46482483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.46256206 +0000 UTC m=+2860.835188808" watchObservedRunningTime="2026-10-30 22:30:03.46482483 +0000 UTC m=+2860.837451569" Oct 30 22:30:03.576181 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:03.576126 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" podStartSLOduration=3.576108209 podStartE2EDuration="3.576108209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.574979547 +0000 UTC m=+2860.947606292" watchObservedRunningTime="2026-10-30 22:30:03.576108209 +0000 UTC m=+2860.948734957" Oct 30 22:30:03.692220 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:03.692170 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" podStartSLOduration=3.692157214 podStartE2EDuration="3.692157214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.689527502 +0000 UTC m=+2861.062154248" watchObservedRunningTime="2026-10-30 22:30:03.692157214 +0000 UTC m=+2861.064783959" Oct 30 22:30:03.806053 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:03.805995 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" podStartSLOduration=3.8059822949999997 podStartE2EDuration="3.805982295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.802236078 +0000 UTC m=+2861.174862823" watchObservedRunningTime="2026-10-30 22:30:03.805982295 +0000 UTC m=+2861.178609040" Oct 30 22:30:03.918699 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:03.917533 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" podStartSLOduration=3.917514091 podStartE2EDuration="3.917514091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.914624437 +0000 UTC m=+2861.287251195" watchObservedRunningTime="2026-10-30 22:30:03.917514091 +0000 UTC m=+2861.290140841" Oct 30 22:30:04.358968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.358910 4859 generic.go:356] "Generic (PLEG): container finished" podID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerID="23122602406b47c0b60eda601d045ca684e0070b1b255b76768235600f6c6fc4" exitCode=60 Oct 30 22:30:04.359429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.358989 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" event={"ID":"48bea4c7-cd12-4ed9-ae40-5040e655e562","Type":"ContainerDied","Data":"23122602406b47c0b60eda601d045ca684e0070b1b255b76768235600f6c6fc4"} Oct 30 22:30:04.359429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.359055 4859 scope.go:122] "RemoveContainer" containerID="381dd823d77468b677c12577fc547cde00aa2c8ada12d2c6da2558e08f0a345b" Oct 30 22:30:04.359429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.359309 4859 scope.go:122] "RemoveContainer" containerID="23122602406b47c0b60eda601d045ca684e0070b1b255b76768235600f6c6fc4" Oct 30 22:30:04.359632 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:04.359598 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb_odh-ai-gateway-infra(48bea4c7-cd12-4ed9-ae40-5040e655e562)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" Oct 30 22:30:04.361723 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.361699 4859 generic.go:356] "Generic (PLEG): container finished" podID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerID="d56b41c243e2c4a84d55fb2268da45ff7f67f5b8fa54e5c142cdb77aa78a87db" exitCode=60 Oct 30 22:30:04.361995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.361968 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" event={"ID":"13f89a58-0eaf-4da2-b9f4-684618c3187f","Type":"ContainerDied","Data":"d56b41c243e2c4a84d55fb2268da45ff7f67f5b8fa54e5c142cdb77aa78a87db"} Oct 30 22:30:04.362152 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.362136 4859 scope.go:122] "RemoveContainer" containerID="d56b41c243e2c4a84d55fb2268da45ff7f67f5b8fa54e5c142cdb77aa78a87db" Oct 30 22:30:04.362444 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:04.362421 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8_odh-ai-gateway-infra(13f89a58-0eaf-4da2-b9f4-684618c3187f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" Oct 30 22:30:04.364087 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.364065 4859 generic.go:356] "Generic (PLEG): container finished" podID="c50f5739-92bc-4948-bc40-17e904811f40" containerID="495ab785c240ba74c289d6fd4ec083c66cb35cbf589647c04d023c057569f3e3" exitCode=60 Oct 30 22:30:04.364170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.364142 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" event={"ID":"c50f5739-92bc-4948-bc40-17e904811f40","Type":"ContainerDied","Data":"495ab785c240ba74c289d6fd4ec083c66cb35cbf589647c04d023c057569f3e3"} Oct 30 22:30:04.364372 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.364355 4859 scope.go:122] "RemoveContainer" containerID="495ab785c240ba74c289d6fd4ec083c66cb35cbf589647c04d023c057569f3e3" Oct 30 22:30:04.364616 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:04.364599 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx_odh-ai-gateway-infra(c50f5739-92bc-4948-bc40-17e904811f40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" podUID="c50f5739-92bc-4948-bc40-17e904811f40" Oct 30 22:30:04.366397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.366376 4859 generic.go:356] "Generic (PLEG): container finished" podID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerID="6f3ca6ee307863462a879d533b661109907038d3709af2e2c4819ac4326a6a87" exitCode=60 Oct 30 22:30:04.366501 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.366454 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" event={"ID":"e5f5f4fb-7141-43d0-be2c-3217a7e45f84","Type":"ContainerDied","Data":"6f3ca6ee307863462a879d533b661109907038d3709af2e2c4819ac4326a6a87"} Oct 30 22:30:04.366619 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.366605 4859 scope.go:122] "RemoveContainer" containerID="6f3ca6ee307863462a879d533b661109907038d3709af2e2c4819ac4326a6a87" Oct 30 22:30:04.366839 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:04.366817 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9_odh-ai-gateway-infra(e5f5f4fb-7141-43d0-be2c-3217a7e45f84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" Oct 30 22:30:04.368490 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.368461 4859 generic.go:356] "Generic (PLEG): container finished" podID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerID="f5e5a61bf521fc1fc8ad6f1cac97787db82a35a3c3cc167205b191e7e7f34151" exitCode=60 Oct 30 22:30:04.368558 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.368507 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" event={"ID":"4ea61988-fb62-4427-aea8-5e7b5360ada5","Type":"ContainerDied","Data":"f5e5a61bf521fc1fc8ad6f1cac97787db82a35a3c3cc167205b191e7e7f34151"} Oct 30 22:30:04.368715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.368702 4859 scope.go:122] "RemoveContainer" containerID="f5e5a61bf521fc1fc8ad6f1cac97787db82a35a3c3cc167205b191e7e7f34151" Oct 30 22:30:04.368902 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:04.368886 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d_odh-ai-gateway-infra(4ea61988-fb62-4427-aea8-5e7b5360ada5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" Oct 30 22:30:04.387650 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.387627 4859 scope.go:122] "RemoveContainer" containerID="b2cd9fb04bab8b1d0f2d84da841f9339043cd857ec5c48f4d2a0a85f82a169f1" Oct 30 22:30:04.407373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.407351 4859 scope.go:122] "RemoveContainer" containerID="336048c1d21f81ce25a797dd949b6cb09c62005472dc2b9b945b15b2e88bad5f" Oct 30 22:30:04.419721 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.419704 4859 scope.go:122] "RemoveContainer" containerID="79b9198d3ff32e52b5688d0b4b4985ce86301716e7461af36375cd426fc4d4da" Oct 30 22:30:04.431857 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:04.431839 4859 scope.go:122] "RemoveContainer" containerID="225d60cf4d81e4e5f146c89eaf8bba63f03a84217f36feeb127ff75bc076814d" Oct 30 22:30:05.377255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:05.377225 4859 scope.go:122] "RemoveContainer" containerID="23122602406b47c0b60eda601d045ca684e0070b1b255b76768235600f6c6fc4" Oct 30 22:30:05.377735 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:05.377536 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb_odh-ai-gateway-infra(48bea4c7-cd12-4ed9-ae40-5040e655e562)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" Oct 30 22:30:05.379545 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:05.379521 4859 scope.go:122] "RemoveContainer" containerID="d56b41c243e2c4a84d55fb2268da45ff7f67f5b8fa54e5c142cdb77aa78a87db" Oct 30 22:30:05.379802 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:05.379778 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8_odh-ai-gateway-infra(13f89a58-0eaf-4da2-b9f4-684618c3187f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" Oct 30 22:30:05.381179 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:05.381157 4859 scope.go:122] "RemoveContainer" containerID="495ab785c240ba74c289d6fd4ec083c66cb35cbf589647c04d023c057569f3e3" Oct 30 22:30:05.381407 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:05.381379 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx_odh-ai-gateway-infra(c50f5739-92bc-4948-bc40-17e904811f40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" podUID="c50f5739-92bc-4948-bc40-17e904811f40" Oct 30 22:30:05.382692 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:05.382673 4859 scope.go:122] "RemoveContainer" containerID="6f3ca6ee307863462a879d533b661109907038d3709af2e2c4819ac4326a6a87" Oct 30 22:30:05.382919 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:05.382896 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9_odh-ai-gateway-infra(e5f5f4fb-7141-43d0-be2c-3217a7e45f84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" Oct 30 22:30:05.384202 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:05.384186 4859 scope.go:122] "RemoveContainer" containerID="f5e5a61bf521fc1fc8ad6f1cac97787db82a35a3c3cc167205b191e7e7f34151" Oct 30 22:30:05.384358 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:05.384342 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d_odh-ai-gateway-infra(4ea61988-fb62-4427-aea8-5e7b5360ada5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" Oct 30 22:30:09.878912 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:09.878864 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:30:14.734057 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:14.734022 4859 scope.go:122] "RemoveContainer" containerID="6f3ca6ee307863462a879d533b661109907038d3709af2e2c4819ac4326a6a87" Oct 30 22:30:14.734538 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:14.734170 4859 scope.go:122] "RemoveContainer" containerID="495ab785c240ba74c289d6fd4ec083c66cb35cbf589647c04d023c057569f3e3" Oct 30 22:30:14.734538 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:14.734281 4859 scope.go:122] "RemoveContainer" containerID="f5e5a61bf521fc1fc8ad6f1cac97787db82a35a3c3cc167205b191e7e7f34151" Oct 30 22:30:15.508805 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:15.508770 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" event={"ID":"e5f5f4fb-7141-43d0-be2c-3217a7e45f84","Type":"ContainerStarted","Data":"c75fbe8fbb2fc5422fae048bf70e289344f1d31cf16111ec8114a13d5dd30c0f"} Oct 30 22:30:15.510900 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:15.510869 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" event={"ID":"4ea61988-fb62-4427-aea8-5e7b5360ada5","Type":"ContainerStarted","Data":"63561be78092f027325cf80f1481b1b457ce0cb087efce79273c8cb795b48765"} Oct 30 22:30:15.512795 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:15.512776 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" event={"ID":"c50f5739-92bc-4948-bc40-17e904811f40","Type":"ContainerStarted","Data":"1d5f126294dd0ed482570afda89a4443f20f02746a47ed087274d3c73940d849"} Oct 30 22:30:15.734113 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:15.734087 4859 scope.go:122] "RemoveContainer" containerID="d56b41c243e2c4a84d55fb2268da45ff7f67f5b8fa54e5c142cdb77aa78a87db" Oct 30 22:30:16.522364 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.522329 4859 generic.go:356] "Generic (PLEG): container finished" podID="c50f5739-92bc-4948-bc40-17e904811f40" containerID="1d5f126294dd0ed482570afda89a4443f20f02746a47ed087274d3c73940d849" exitCode=60 Oct 30 22:30:16.522536 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.522371 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" event={"ID":"c50f5739-92bc-4948-bc40-17e904811f40","Type":"ContainerDied","Data":"1d5f126294dd0ed482570afda89a4443f20f02746a47ed087274d3c73940d849"} Oct 30 22:30:16.522536 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.522431 4859 scope.go:122] "RemoveContainer" containerID="495ab785c240ba74c289d6fd4ec083c66cb35cbf589647c04d023c057569f3e3" Oct 30 22:30:16.522751 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.522730 4859 scope.go:122] "RemoveContainer" containerID="1d5f126294dd0ed482570afda89a4443f20f02746a47ed087274d3c73940d849" Oct 30 22:30:16.523085 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:16.523054 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx_odh-ai-gateway-infra(c50f5739-92bc-4948-bc40-17e904811f40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" podUID="c50f5739-92bc-4948-bc40-17e904811f40" Oct 30 22:30:16.524473 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.524455 4859 generic.go:356] "Generic (PLEG): container finished" podID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerID="c75fbe8fbb2fc5422fae048bf70e289344f1d31cf16111ec8114a13d5dd30c0f" exitCode=60 Oct 30 22:30:16.524557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.524518 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" event={"ID":"e5f5f4fb-7141-43d0-be2c-3217a7e45f84","Type":"ContainerDied","Data":"c75fbe8fbb2fc5422fae048bf70e289344f1d31cf16111ec8114a13d5dd30c0f"} Oct 30 22:30:16.524822 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.524806 4859 scope.go:122] "RemoveContainer" containerID="c75fbe8fbb2fc5422fae048bf70e289344f1d31cf16111ec8114a13d5dd30c0f" Oct 30 22:30:16.525058 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:16.525038 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9_odh-ai-gateway-infra(e5f5f4fb-7141-43d0-be2c-3217a7e45f84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" Oct 30 22:30:16.526321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.526299 4859 generic.go:356] "Generic (PLEG): container finished" podID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerID="63561be78092f027325cf80f1481b1b457ce0cb087efce79273c8cb795b48765" exitCode=60 Oct 30 22:30:16.526414 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.526363 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" event={"ID":"4ea61988-fb62-4427-aea8-5e7b5360ada5","Type":"ContainerDied","Data":"63561be78092f027325cf80f1481b1b457ce0cb087efce79273c8cb795b48765"} Oct 30 22:30:16.526656 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.526637 4859 scope.go:122] "RemoveContainer" containerID="63561be78092f027325cf80f1481b1b457ce0cb087efce79273c8cb795b48765" Oct 30 22:30:16.526888 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:16.526865 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d_odh-ai-gateway-infra(4ea61988-fb62-4427-aea8-5e7b5360ada5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" Oct 30 22:30:16.528648 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.528628 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" event={"ID":"13f89a58-0eaf-4da2-b9f4-684618c3187f","Type":"ContainerStarted","Data":"5d62afd8e28d12eb2ca764c2aa4e39461db23355da03c51d74d1bd418674f7a0"} Oct 30 22:30:16.535374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.535357 4859 scope.go:122] "RemoveContainer" containerID="6f3ca6ee307863462a879d533b661109907038d3709af2e2c4819ac4326a6a87" Oct 30 22:30:16.547816 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.547792 4859 scope.go:122] "RemoveContainer" containerID="f5e5a61bf521fc1fc8ad6f1cac97787db82a35a3c3cc167205b191e7e7f34151" Oct 30 22:30:16.645557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.645531 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9"] Oct 30 22:30:16.755664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.755634 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx"] Oct 30 22:30:16.863539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:16.863500 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d"] Oct 30 22:30:17.574861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.574667 4859 generic.go:356] "Generic (PLEG): container finished" podID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerID="5d62afd8e28d12eb2ca764c2aa4e39461db23355da03c51d74d1bd418674f7a0" exitCode=60 Oct 30 22:30:17.574861 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.574762 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" event={"ID":"13f89a58-0eaf-4da2-b9f4-684618c3187f","Type":"ContainerDied","Data":"5d62afd8e28d12eb2ca764c2aa4e39461db23355da03c51d74d1bd418674f7a0"} Oct 30 22:30:17.575205 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.575185 4859 scope.go:122] "RemoveContainer" containerID="d56b41c243e2c4a84d55fb2268da45ff7f67f5b8fa54e5c142cdb77aa78a87db" Oct 30 22:30:17.575895 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.575871 4859 scope.go:122] "RemoveContainer" containerID="5d62afd8e28d12eb2ca764c2aa4e39461db23355da03c51d74d1bd418674f7a0" Oct 30 22:30:17.576418 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:17.576274 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8_odh-ai-gateway-infra(13f89a58-0eaf-4da2-b9f4-684618c3187f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" Oct 30 22:30:17.734453 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.734427 4859 scope.go:122] "RemoveContainer" containerID="23122602406b47c0b60eda601d045ca684e0070b1b255b76768235600f6c6fc4" Oct 30 22:30:17.760789 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.760762 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" Oct 30 22:30:17.805273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.805252 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50f5739-92bc-4948-bc40-17e904811f40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50f5739-92bc-4948-bc40-17e904811f40-maas-api-service-ca\") pod \"c50f5739-92bc-4948-bc40-17e904811f40\" (UID: \"c50f5739-92bc-4948-bc40-17e904811f40\") " Oct 30 22:30:17.805384 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.805338 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50f5739-92bc-4948-bc40-17e904811f40-kube-api-access-mfsdn\" (UniqueName: \"kubernetes.io/projected/c50f5739-92bc-4948-bc40-17e904811f40-kube-api-access-mfsdn\") pod \"c50f5739-92bc-4948-bc40-17e904811f40\" (UID: \"c50f5739-92bc-4948-bc40-17e904811f40\") " Oct 30 22:30:17.805652 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.805624 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50f5739-92bc-4948-bc40-17e904811f40-maas-api-service-ca" pod "c50f5739-92bc-4948-bc40-17e904811f40" (UID: "c50f5739-92bc-4948-bc40-17e904811f40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:17.805805 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.805784 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50f5739-92bc-4948-bc40-17e904811f40-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.807453 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.807425 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50f5739-92bc-4948-bc40-17e904811f40-kube-api-access-mfsdn" pod "c50f5739-92bc-4948-bc40-17e904811f40" (UID: "c50f5739-92bc-4948-bc40-17e904811f40"). InnerVolumeSpecName "kube-api-access-mfsdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:17.824780 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.824759 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" Oct 30 22:30:17.827103 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.827058 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" Oct 30 22:30:17.876235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.876210 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8"] Oct 30 22:30:17.906688 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.906659 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-kube-api-access-ml2xz\" (UniqueName: \"kubernetes.io/projected/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-kube-api-access-ml2xz\") pod \"e5f5f4fb-7141-43d0-be2c-3217a7e45f84\" (UID: \"e5f5f4fb-7141-43d0-be2c-3217a7e45f84\") " Oct 30 22:30:17.906807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.906739 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea61988-fb62-4427-aea8-5e7b5360ada5-kube-api-access-lwkjt\" (UniqueName: \"kubernetes.io/projected/4ea61988-fb62-4427-aea8-5e7b5360ada5-kube-api-access-lwkjt\") pod \"4ea61988-fb62-4427-aea8-5e7b5360ada5\" (UID: \"4ea61988-fb62-4427-aea8-5e7b5360ada5\") " Oct 30 22:30:17.906807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.906783 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea61988-fb62-4427-aea8-5e7b5360ada5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea61988-fb62-4427-aea8-5e7b5360ada5-maas-api-service-ca\") pod \"4ea61988-fb62-4427-aea8-5e7b5360ada5\" (UID: \"4ea61988-fb62-4427-aea8-5e7b5360ada5\") " Oct 30 22:30:17.906969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.906927 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-maas-api-service-ca\") pod \"e5f5f4fb-7141-43d0-be2c-3217a7e45f84\" (UID: \"e5f5f4fb-7141-43d0-be2c-3217a7e45f84\") " Oct 30 22:30:17.907229 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.907204 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea61988-fb62-4427-aea8-5e7b5360ada5-maas-api-service-ca" pod "4ea61988-fb62-4427-aea8-5e7b5360ada5" (UID: "4ea61988-fb62-4427-aea8-5e7b5360ada5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:17.907293 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.907229 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-maas-api-service-ca" pod "e5f5f4fb-7141-43d0-be2c-3217a7e45f84" (UID: "e5f5f4fb-7141-43d0-be2c-3217a7e45f84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:17.907389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.907373 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.907440 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.907396 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfsdn\" (UniqueName: \"kubernetes.io/projected/c50f5739-92bc-4948-bc40-17e904811f40-kube-api-access-mfsdn\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.907440 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.907413 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea61988-fb62-4427-aea8-5e7b5360ada5-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.908608 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.908583 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-kube-api-access-ml2xz" pod "e5f5f4fb-7141-43d0-be2c-3217a7e45f84" (UID: "e5f5f4fb-7141-43d0-be2c-3217a7e45f84"). InnerVolumeSpecName "kube-api-access-ml2xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:17.908733 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:17.908715 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea61988-fb62-4427-aea8-5e7b5360ada5-kube-api-access-lwkjt" pod "4ea61988-fb62-4427-aea8-5e7b5360ada5" (UID: "4ea61988-fb62-4427-aea8-5e7b5360ada5"). InnerVolumeSpecName "kube-api-access-lwkjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:18.008332 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.008308 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml2xz\" (UniqueName: \"kubernetes.io/projected/e5f5f4fb-7141-43d0-be2c-3217a7e45f84-kube-api-access-ml2xz\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.008332 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.008331 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwkjt\" (UniqueName: \"kubernetes.io/projected/4ea61988-fb62-4427-aea8-5e7b5360ada5-kube-api-access-lwkjt\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.591455 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.591423 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" event={"ID":"e5f5f4fb-7141-43d0-be2c-3217a7e45f84","Type":"ContainerDied","Data":"9c4f2cc98f64f57071d8d41c23ffccf3ecfee1f30dfe1f383fa22aad003d090b"} Oct 30 22:30:18.591455 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.591442 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9" Oct 30 22:30:18.591685 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.591497 4859 scope.go:122] "RemoveContainer" containerID="c75fbe8fbb2fc5422fae048bf70e289344f1d31cf16111ec8114a13d5dd30c0f" Oct 30 22:30:18.593727 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.593703 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" event={"ID":"4ea61988-fb62-4427-aea8-5e7b5360ada5","Type":"ContainerDied","Data":"56c5efc9829fa1a612e50ef40308184d5abdaf3674e355fc712880ec995b341d"} Oct 30 22:30:18.594199 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.594003 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d" Oct 30 22:30:18.600517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.600493 4859 generic.go:356] "Generic (PLEG): container finished" podID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerID="7076c70ab1cd7f11687d9be426ef9722522400e54a657c441f0db4b7eb9e483d" exitCode=60 Oct 30 22:30:18.600599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.600564 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" event={"ID":"48bea4c7-cd12-4ed9-ae40-5040e655e562","Type":"ContainerDied","Data":"7076c70ab1cd7f11687d9be426ef9722522400e54a657c441f0db4b7eb9e483d"} Oct 30 22:30:18.601000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.600982 4859 scope.go:122] "RemoveContainer" containerID="7076c70ab1cd7f11687d9be426ef9722522400e54a657c441f0db4b7eb9e483d" Oct 30 22:30:18.601297 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:18.601270 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb_odh-ai-gateway-infra(48bea4c7-cd12-4ed9-ae40-5040e655e562)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" Oct 30 22:30:18.603923 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.603901 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" event={"ID":"c50f5739-92bc-4948-bc40-17e904811f40","Type":"ContainerDied","Data":"8da8ba621c115aa636f2a7137f4635d0afdc6239fb4a69e6b998ef1dfa4acebd"} Oct 30 22:30:18.604039 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.603924 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx" Oct 30 22:30:18.616628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.616605 4859 scope.go:122] "RemoveContainer" containerID="63561be78092f027325cf80f1481b1b457ce0cb087efce79273c8cb795b48765" Oct 30 22:30:18.626464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.626446 4859 scope.go:122] "RemoveContainer" containerID="23122602406b47c0b60eda601d045ca684e0070b1b255b76768235600f6c6fc4" Oct 30 22:30:18.640365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.640345 4859 scope.go:122] "RemoveContainer" containerID="1d5f126294dd0ed482570afda89a4443f20f02746a47ed087274d3c73940d849" Oct 30 22:30:18.758774 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.758752 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" Oct 30 22:30:18.827573 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.827502 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f89a58-0eaf-4da2-b9f4-684618c3187f-kube-api-access-tzfbd\" (UniqueName: \"kubernetes.io/projected/13f89a58-0eaf-4da2-b9f4-684618c3187f-kube-api-access-tzfbd\") pod \"13f89a58-0eaf-4da2-b9f4-684618c3187f\" (UID: \"13f89a58-0eaf-4da2-b9f4-684618c3187f\") " Oct 30 22:30:18.827573 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.827547 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f89a58-0eaf-4da2-b9f4-684618c3187f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f89a58-0eaf-4da2-b9f4-684618c3187f-maas-api-service-ca\") pod \"13f89a58-0eaf-4da2-b9f4-684618c3187f\" (UID: \"13f89a58-0eaf-4da2-b9f4-684618c3187f\") " Oct 30 22:30:18.828043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.827909 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f89a58-0eaf-4da2-b9f4-684618c3187f-maas-api-service-ca" pod "13f89a58-0eaf-4da2-b9f4-684618c3187f" (UID: "13f89a58-0eaf-4da2-b9f4-684618c3187f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:18.828130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.828111 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f89a58-0eaf-4da2-b9f4-684618c3187f-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.829793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.829774 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f89a58-0eaf-4da2-b9f4-684618c3187f-kube-api-access-tzfbd" pod "13f89a58-0eaf-4da2-b9f4-684618c3187f" (UID: "13f89a58-0eaf-4da2-b9f4-684618c3187f"). InnerVolumeSpecName "kube-api-access-tzfbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:18.867685 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.867665 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9"] Oct 30 22:30:18.873719 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.873694 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-086575-29889990-t2sh9"] Oct 30 22:30:18.929819 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:18.929787 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfbd\" (UniqueName: \"kubernetes.io/projected/13f89a58-0eaf-4da2-b9f4-684618c3187f-kube-api-access-tzfbd\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.099237 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.099204 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx"] Oct 30 22:30:19.105050 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.105013 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889990-smkcx"] Oct 30 22:30:19.223371 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.223340 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d"] Oct 30 22:30:19.237627 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.237594 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889990-j9r4d"] Oct 30 22:30:19.616058 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.616016 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" event={"ID":"13f89a58-0eaf-4da2-b9f4-684618c3187f","Type":"ContainerDied","Data":"57bc4bbba4d897a1b276e1b48231647101c390395b85b9c63472084cef15d8a9"} Oct 30 22:30:19.616262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.616091 4859 scope.go:122] "RemoveContainer" containerID="5d62afd8e28d12eb2ca764c2aa4e39461db23355da03c51d74d1bd418674f7a0" Oct 30 22:30:19.616262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.616032 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8" Oct 30 22:30:19.768519 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.768493 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb"] Oct 30 22:30:19.781088 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.781063 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8"] Oct 30 22:30:19.817695 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.817669 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-37a390-29889990-s4jn8"] Oct 30 22:30:19.925852 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:19.925829 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" Oct 30 22:30:20.044249 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.044216 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bea4c7-cd12-4ed9-ae40-5040e655e562-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bea4c7-cd12-4ed9-ae40-5040e655e562-maas-api-service-ca\") pod \"48bea4c7-cd12-4ed9-ae40-5040e655e562\" (UID: \"48bea4c7-cd12-4ed9-ae40-5040e655e562\") " Oct 30 22:30:20.044617 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.044383 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bea4c7-cd12-4ed9-ae40-5040e655e562-kube-api-access-sdt8b\" (UniqueName: \"kubernetes.io/projected/48bea4c7-cd12-4ed9-ae40-5040e655e562-kube-api-access-sdt8b\") pod \"48bea4c7-cd12-4ed9-ae40-5040e655e562\" (UID: \"48bea4c7-cd12-4ed9-ae40-5040e655e562\") " Oct 30 22:30:20.044707 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.044617 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bea4c7-cd12-4ed9-ae40-5040e655e562-maas-api-service-ca" pod "48bea4c7-cd12-4ed9-ae40-5040e655e562" (UID: "48bea4c7-cd12-4ed9-ae40-5040e655e562"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:20.044924 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.044906 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bea4c7-cd12-4ed9-ae40-5040e655e562-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.046581 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.046558 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bea4c7-cd12-4ed9-ae40-5040e655e562-kube-api-access-sdt8b" pod "48bea4c7-cd12-4ed9-ae40-5040e655e562" (UID: "48bea4c7-cd12-4ed9-ae40-5040e655e562"). InnerVolumeSpecName "kube-api-access-sdt8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:20.146553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.146514 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdt8b\" (UniqueName: \"kubernetes.io/projected/48bea4c7-cd12-4ed9-ae40-5040e655e562-kube-api-access-sdt8b\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.634505 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.634467 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" Oct 30 22:30:20.634733 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.634651 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb" event={"ID":"48bea4c7-cd12-4ed9-ae40-5040e655e562","Type":"ContainerDied","Data":"5b848771d2fe34f4eb9c4867b007aa558d51ff67ca3156ae30015b3971dba2fc"} Oct 30 22:30:20.634809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.634776 4859 scope.go:122] "RemoveContainer" containerID="7076c70ab1cd7f11687d9be426ef9722522400e54a657c441f0db4b7eb9e483d" Oct 30 22:30:20.741543 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.741514 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" path="/var/lib/kubelet/pods/13f89a58-0eaf-4da2-b9f4-684618c3187f/volumes" Oct 30 22:30:20.742040 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.742019 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" path="/var/lib/kubelet/pods/4ea61988-fb62-4427-aea8-5e7b5360ada5/volumes" Oct 30 22:30:20.742463 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.742444 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50f5739-92bc-4948-bc40-17e904811f40" path="/var/lib/kubelet/pods/c50f5739-92bc-4948-bc40-17e904811f40/volumes" Oct 30 22:30:20.743203 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.743174 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" path="/var/lib/kubelet/pods/e5f5f4fb-7141-43d0-be2c-3217a7e45f84/volumes" Oct 30 22:30:20.813200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.813175 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb"] Oct 30 22:30:20.862411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:20.862391 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889990-k2xrb"] Oct 30 22:30:22.742629 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:22.742598 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" path="/var/lib/kubelet/pods/48bea4c7-cd12-4ed9-ae40-5040e655e562/volumes" Oct 30 22:30:23.484374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484339 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg"] Oct 30 22:30:23.484869 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484855 4859 cpu_manager.go:414] "containerMap: removing container" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerName="cleanup" Oct 30 22:30:23.484916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484870 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerName="cleanup" Oct 30 22:30:23.484916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484881 4859 cpu_manager.go:414] "containerMap: removing container" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerName="cleanup" Oct 30 22:30:23.484916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484885 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerName="cleanup" Oct 30 22:30:23.484916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484893 4859 cpu_manager.go:414] "containerMap: removing container" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerName="cleanup" Oct 30 22:30:23.484916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484898 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerName="cleanup" Oct 30 22:30:23.484916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484907 4859 cpu_manager.go:414] "containerMap: removing container" podUID="c50f5739-92bc-4948-bc40-17e904811f40" containerName="cleanup" Oct 30 22:30:23.484916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484911 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50f5739-92bc-4948-bc40-17e904811f40" containerName="cleanup" Oct 30 22:30:23.484916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484917 4859 cpu_manager.go:414] "containerMap: removing container" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484921 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484932 4859 cpu_manager.go:414] "containerMap: removing container" podUID="c50f5739-92bc-4948-bc40-17e904811f40" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484937 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50f5739-92bc-4948-bc40-17e904811f40" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484968 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484974 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484983 4859 cpu_manager.go:414] "containerMap: removing container" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484987 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484995 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.484999 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485006 4859 cpu_manager.go:414] "containerMap: removing container" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485011 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485026 4859 cpu_manager.go:414] "containerMap: removing container" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485030 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485117 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485126 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485133 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485138 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485145 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485150 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485156 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50f5739-92bc-4948-bc40-17e904811f40" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485163 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485169 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485173 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485179 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50f5739-92bc-4948-bc40-17e904811f40" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485183 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerName="cleanup" Oct 30 22:30:23.485188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.485189 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50f5739-92bc-4948-bc40-17e904811f40" containerName="cleanup" Oct 30 22:30:23.488368 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.488353 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.542902 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.542879 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a0116e-openshift-default-dockercfg-2tvzm\"" Oct 30 22:30:23.582004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.581939 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-credential-socket\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.582004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.582010 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-socket\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.582194 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.582165 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-token\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.582233 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.582210 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-envoy\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.582314 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.582292 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istiod-ca-cert\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.582394 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.582380 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-data\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.582446 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.582434 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqvr\" (UniqueName: \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-kube-api-access-tmqvr\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.582480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.582455 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-certs\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.582514 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.582487 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-podinfo\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.595517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.595495 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg"] Oct 30 22:30:23.683482 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.683445 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-podinfo\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.683655 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.683530 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-credential-socket\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.683655 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.683555 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-socket\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.683655 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.683620 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-token\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.683655 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.683643 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-envoy\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.683875 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.683668 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istiod-ca-cert\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.684036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.683986 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-data\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.684404 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.684378 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-socket\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.684581 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.684458 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-data\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.684581 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.683993 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-credential-socket\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.684837 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.684686 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istiod-ca-cert\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.684902 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.684828 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqvr\" (UniqueName: \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-kube-api-access-tmqvr\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.684902 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.684858 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-certs\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.685226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.685207 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-certs\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.686687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.686670 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-podinfo\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.687128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.687110 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-envoy\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.741509 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.741454 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-token\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.743363 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.743340 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqvr\" (UniqueName: \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-kube-api-access-tmqvr\") pod \"e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:23.801490 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:23.801465 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:24.373504 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:24.373475 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg"] Oct 30 22:30:24.411398 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:30:24.411373 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3efccd2_b4f7_47ad_ac24_326f5246aac6.slice/crio-3c5b65c94a5355756377af5f587fc7a3d211fb66f3fa2d53e55056b8d30366f3 WatchSource:0}: Error finding container 3c5b65c94a5355756377af5f587fc7a3d211fb66f3fa2d53e55056b8d30366f3: Status 404 returned error can't find the container with id 3c5b65c94a5355756377af5f587fc7a3d211fb66f3fa2d53e55056b8d30366f3 Oct 30 22:30:24.413665 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:24.413629 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:30:24.413759 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:24.413710 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:30:24.413816 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:24.413756 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:30:24.679428 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:24.679326 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" event={"ID":"e3efccd2-b4f7-47ad-ac24-326f5246aac6","Type":"ContainerStarted","Data":"efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98"} Oct 30 22:30:24.679428 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:24.679409 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" event={"ID":"e3efccd2-b4f7-47ad-ac24-326f5246aac6","Type":"ContainerStarted","Data":"3c5b65c94a5355756377af5f587fc7a3d211fb66f3fa2d53e55056b8d30366f3"} Oct 30 22:30:24.798796 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:24.796989 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" podStartSLOduration=1.796971748 podStartE2EDuration="1.796971748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:24.794804131 +0000 UTC m=+2882.167430876" watchObservedRunningTime="2026-10-30 22:30:24.796971748 +0000 UTC m=+2882.169598494" Oct 30 22:30:24.801588 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:24.801554 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:25.807231 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:25.807200 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:26.692072 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:26.692048 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:26.693123 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:26.693102 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:30:33.883451 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:33.883411 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:30:36.276992 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:36.268045 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:30:36.276992 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:36.268133 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:30:39.884796 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:30:39.884752 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:30:54.236931 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:54.235793 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv"] Oct 30 22:30:54.236931 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:54.236066 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" containerID="cri-o://55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a" gracePeriod=30 Oct 30 22:30:56.436558 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:56.436522 4859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4bfdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:30:56.436929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:56.436598 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:30:59.525389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.525365 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:30:59.619166 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619135 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-data\") pod \"c98bbb81-f4dc-493d-92c7-d43984c30517\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " Oct 30 22:30:59.619320 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619183 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-credential-socket\") pod \"c98bbb81-f4dc-493d-92c7-d43984c30517\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " Oct 30 22:30:59.619320 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619241 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-socket\") pod \"c98bbb81-f4dc-493d-92c7-d43984c30517\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " Oct 30 22:30:59.619320 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619307 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-podinfo\") pod \"c98bbb81-f4dc-493d-92c7-d43984c30517\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " Oct 30 22:30:59.619450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619334 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98bbb81-f4dc-493d-92c7-d43984c30517-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98bbb81-f4dc-493d-92c7-d43984c30517-istiod-ca-cert\") pod \"c98bbb81-f4dc-493d-92c7-d43984c30517\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " Oct 30 22:30:59.619450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619390 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-token\" (UniqueName: \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-token\") pod \"c98bbb81-f4dc-493d-92c7-d43984c30517\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " Oct 30 22:30:59.619450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619417 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-credential-socket" pod "c98bbb81-f4dc-493d-92c7-d43984c30517" (UID: "c98bbb81-f4dc-493d-92c7-d43984c30517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:59.619450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619427 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-data" pod "c98bbb81-f4dc-493d-92c7-d43984c30517" (UID: "c98bbb81-f4dc-493d-92c7-d43984c30517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:59.619649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619423 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-kube-api-access-hts8n\" (UniqueName: \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-kube-api-access-hts8n\") pod \"c98bbb81-f4dc-493d-92c7-d43984c30517\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " Oct 30 22:30:59.619649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619538 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-certs\") pod \"c98bbb81-f4dc-493d-92c7-d43984c30517\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " Oct 30 22:30:59.619649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619587 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-envoy\") pod \"c98bbb81-f4dc-493d-92c7-d43984c30517\" (UID: \"c98bbb81-f4dc-493d-92c7-d43984c30517\") " Oct 30 22:30:59.619649 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619589 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-socket" pod "c98bbb81-f4dc-493d-92c7-d43984c30517" (UID: "c98bbb81-f4dc-493d-92c7-d43984c30517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:59.620369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619796 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98bbb81-f4dc-493d-92c7-d43984c30517-istiod-ca-cert" pod "c98bbb81-f4dc-493d-92c7-d43984c30517" (UID: "c98bbb81-f4dc-493d-92c7-d43984c30517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:59.620369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.619998 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-certs" pod "c98bbb81-f4dc-493d-92c7-d43984c30517" (UID: "c98bbb81-f4dc-493d-92c7-d43984c30517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:59.620369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.620270 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.620369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.620291 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.620369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.620310 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.620369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.620327 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98bbb81-f4dc-493d-92c7-d43984c30517-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.620369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.620340 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.621639 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.621616 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-token" pod "c98bbb81-f4dc-493d-92c7-d43984c30517" (UID: "c98bbb81-f4dc-493d-92c7-d43984c30517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:59.621839 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.621818 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-podinfo" pod "c98bbb81-f4dc-493d-92c7-d43984c30517" (UID: "c98bbb81-f4dc-493d-92c7-d43984c30517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:30:59.621910 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.621840 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-kube-api-access-hts8n" pod "c98bbb81-f4dc-493d-92c7-d43984c30517" (UID: "c98bbb81-f4dc-493d-92c7-d43984c30517"). InnerVolumeSpecName "kube-api-access-hts8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:59.621910 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.621888 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-envoy" pod "c98bbb81-f4dc-493d-92c7-d43984c30517" (UID: "c98bbb81-f4dc-493d-92c7-d43984c30517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:59.721690 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.721658 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.721690 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.721689 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hts8n\" (UniqueName: \"kubernetes.io/projected/c98bbb81-f4dc-493d-92c7-d43984c30517-kube-api-access-hts8n\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.721881 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.721708 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.721881 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:30:59.721716 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98bbb81-f4dc-493d-92c7-d43984c30517-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:00.009081 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:00.009038 4859 generic.go:356] "Generic (PLEG): container finished" podID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerID="55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a" exitCode=0 Oct 30 22:31:00.009258 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:00.009083 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" Oct 30 22:31:00.009258 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:00.009105 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" event={"ID":"c98bbb81-f4dc-493d-92c7-d43984c30517","Type":"ContainerDied","Data":"55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a"} Oct 30 22:31:00.009258 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:00.009171 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv" event={"ID":"c98bbb81-f4dc-493d-92c7-d43984c30517","Type":"ContainerDied","Data":"4665dcef37718ba00e0f39a5b22b52971f6fb4ab5c6af2127fd58f32e7a0f9e9"} Oct 30 22:31:00.009258 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:00.009200 4859 scope.go:122] "RemoveContainer" containerID="55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a" Oct 30 22:31:00.028890 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:00.028870 4859 scope.go:122] "RemoveContainer" containerID="55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a" Oct 30 22:31:00.029240 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:31:00.029217 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a\": container with ID starting with 55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a not found: ID does not exist" containerID="55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a" Oct 30 22:31:00.029318 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:00.029251 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a"} err="failed to get container status \"55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a\": rpc error: code = NotFound desc = could not find container \"55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a\": container with ID starting with 55d5709e728e5799e27ab916b791bb36b54b937ea9935d421f04f633c3fa722a not found: ID does not exist" Oct 30 22:31:00.174082 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:00.174051 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv"] Oct 30 22:31:00.223472 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:00.223432 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4bfdv"] Oct 30 22:31:00.741070 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:00.741035 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" path="/var/lib/kubelet/pods/c98bbb81-f4dc-493d-92c7-d43984c30517/volumes" Oct 30 22:31:03.286954 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:03.286905 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:31:04.934161 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.934121 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp"] Oct 30 22:31:04.934885 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.934860 4859 cpu_manager.go:414] "containerMap: removing container" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" Oct 30 22:31:04.935030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.934886 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" Oct 30 22:31:04.935030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.934906 4859 cpu_manager.go:414] "containerMap: removing container" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerName="cleanup" Oct 30 22:31:04.935030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.934913 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bea4c7-cd12-4ed9-ae40-5040e655e562" containerName="cleanup" Oct 30 22:31:04.935030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.934930 4859 cpu_manager.go:414] "containerMap: removing container" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerName="cleanup" Oct 30 22:31:04.935030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.934938 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea61988-fb62-4427-aea8-5e7b5360ada5" containerName="cleanup" Oct 30 22:31:04.935030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.934967 4859 cpu_manager.go:414] "containerMap: removing container" podUID="c50f5739-92bc-4948-bc40-17e904811f40" containerName="cleanup" Oct 30 22:31:04.935030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.934974 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50f5739-92bc-4948-bc40-17e904811f40" containerName="cleanup" Oct 30 22:31:04.935030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.934996 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerName="cleanup" Oct 30 22:31:04.935030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.935003 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerName="cleanup" Oct 30 22:31:04.935503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.935171 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f5f4fb-7141-43d0-be2c-3217a7e45f84" containerName="cleanup" Oct 30 22:31:04.935503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.935186 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f89a58-0eaf-4da2-b9f4-684618c3187f" containerName="cleanup" Oct 30 22:31:04.935503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.935201 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98bbb81-f4dc-493d-92c7-d43984c30517" containerName="istio-proxy" Oct 30 22:31:04.941504 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:04.941332 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.042489 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.042455 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp"] Oct 30 22:31:05.092495 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.092453 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.092763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.092538 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxjx\" (UniqueName: \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-kube-api-access-vgxjx\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.092763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.092565 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.092763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.092588 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.092763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.092625 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.092763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.092689 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.092763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.092715 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.093874 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.092822 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.093874 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.092857 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.193840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.193760 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.193840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.193795 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194100 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.193857 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194100 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.193874 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194100 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.193996 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194100 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.194048 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxjx\" (UniqueName: \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-kube-api-access-vgxjx\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194100 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.194071 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194100 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.194097 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.194133 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.194268 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.194295 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.194343 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194636 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.194535 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.194847 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.194824 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.196463 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.196441 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.196682 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.196667 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.250976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.250929 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.251198 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.251180 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxjx\" (UniqueName: \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-kube-api-access-vgxjx\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.257000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.256976 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:05.689142 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:31:05.689115 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb147f1b4_2219_4fcf_977b_e6b3b72b7c7b.slice/crio-458aabc28b63c70d8adda78790f0ffe34c10d6a4dfdd92b545fb37dcf751555b WatchSource:0}: Error finding container 458aabc28b63c70d8adda78790f0ffe34c10d6a4dfdd92b545fb37dcf751555b: Status 404 returned error can't find the container with id 458aabc28b63c70d8adda78790f0ffe34c10d6a4dfdd92b545fb37dcf751555b Oct 30 22:31:05.690059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.690033 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp"] Oct 30 22:31:05.691929 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.691897 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:31:05.692009 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.691979 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:31:05.692049 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:05.692012 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:31:06.109219 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:06.109180 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" event={"ID":"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b","Type":"ContainerStarted","Data":"5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1"} Oct 30 22:31:06.109639 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:06.109244 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" event={"ID":"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b","Type":"ContainerStarted","Data":"458aabc28b63c70d8adda78790f0ffe34c10d6a4dfdd92b545fb37dcf751555b"} Oct 30 22:31:06.229733 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:06.228824 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" podStartSLOduration=2.228803257 podStartE2EDuration="2.228803257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:06.226397271 +0000 UTC m=+2923.599024020" watchObservedRunningTime="2026-10-30 22:31:06.228803257 +0000 UTC m=+2923.601430005" Oct 30 22:31:06.257406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:06.257366 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:06.259257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:06.259225 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:31:06.259385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:06.259288 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:31:07.257248 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:07.257214 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:31:07.257631 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:07.257263 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:31:08.257907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:08.257876 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:31:08.258390 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:08.257927 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:31:09.257418 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:09.257383 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:31:09.257593 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:09.257449 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:31:09.906280 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:31:09.906236 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:31:10.257546 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:10.257464 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:31:10.257546 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:10.257524 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:31:11.257801 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:11.257760 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:31:11.258247 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:11.257827 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:31:12.259376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:12.259338 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:31:12.259995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:12.259959 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:31:13.262550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:13.262515 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:13.263044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:13.262982 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:13.263998 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:13.263971 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:31:22.764673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:22.764635 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2"] Oct 30 22:31:22.765220 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:22.764876 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" podUID="1f54f71b-4cc8-4335-a50a-a34cb07e7c52" containerName="maas-api" containerID="cri-o://3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37" gracePeriod=30 Oct 30 22:31:23.093252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.093223 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:31:23.142288 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.142252 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-metrics-certs\") pod \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " Oct 30 22:31:23.142482 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.142353 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-openshift-service-ca\") pod \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " Oct 30 22:31:23.142557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.142504 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-maas-api-tls\") pod \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " Oct 30 22:31:23.142557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.142541 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-kube-api-access-6ggdl\" (UniqueName: \"kubernetes.io/projected/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-kube-api-access-6ggdl\") pod \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\" (UID: \"1f54f71b-4cc8-4335-a50a-a34cb07e7c52\") " Oct 30 22:31:23.142737 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.142709 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-openshift-service-ca" pod "1f54f71b-4cc8-4335-a50a-a34cb07e7c52" (UID: "1f54f71b-4cc8-4335-a50a-a34cb07e7c52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:23.143095 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.143074 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:23.144588 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.144562 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-maas-api-tls" pod "1f54f71b-4cc8-4335-a50a-a34cb07e7c52" (UID: "1f54f71b-4cc8-4335-a50a-a34cb07e7c52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:23.144675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.144587 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-kube-api-access-6ggdl" pod "1f54f71b-4cc8-4335-a50a-a34cb07e7c52" (UID: "1f54f71b-4cc8-4335-a50a-a34cb07e7c52"). InnerVolumeSpecName "kube-api-access-6ggdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:23.144675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.144599 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-metrics-certs" pod "1f54f71b-4cc8-4335-a50a-a34cb07e7c52" (UID: "1f54f71b-4cc8-4335-a50a-a34cb07e7c52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:23.247981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.244883 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:23.247981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.244930 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ggdl\" (UniqueName: \"kubernetes.io/projected/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-kube-api-access-6ggdl\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:23.247981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.244965 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f54f71b-4cc8-4335-a50a-a34cb07e7c52-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:23.250667 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.250643 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" Oct 30 22:31:23.251108 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.251075 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" event={"ID":"1f54f71b-4cc8-4335-a50a-a34cb07e7c52","Type":"ContainerDied","Data":"3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37"} Oct 30 22:31:23.251236 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.251145 4859 scope.go:122] "RemoveContainer" containerID="3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37" Oct 30 22:31:23.263061 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.250576 4859 generic.go:356] "Generic (PLEG): container finished" podID="1f54f71b-4cc8-4335-a50a-a34cb07e7c52" containerID="3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37" exitCode=0 Oct 30 22:31:23.263197 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.263160 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2" event={"ID":"1f54f71b-4cc8-4335-a50a-a34cb07e7c52","Type":"ContainerDied","Data":"73d230fccf5d8f7580e5bc3c8e33652b3027f7d11266c7606fe62aff1e1e82f6"} Oct 30 22:31:23.275972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.275909 4859 scope.go:122] "RemoveContainer" containerID="3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37" Oct 30 22:31:23.276334 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:31:23.276307 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37\": container with ID starting with 3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37 not found: ID does not exist" containerID="3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37" Oct 30 22:31:23.276421 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.276345 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37"} err="failed to get container status \"3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37\": rpc error: code = NotFound desc = could not find container \"3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37\": container with ID starting with 3ef2f0083b840b650529b048ba76bcc452b08b4642a6bf03c28b07f4ba4acc37 not found: ID does not exist" Oct 30 22:31:23.422415 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.422374 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2"] Oct 30 22:31:23.425576 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:23.425548 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b563fd-58956c6b7d-n2sz2"] Oct 30 22:31:24.742587 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:24.742553 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f54f71b-4cc8-4335-a50a-a34cb07e7c52" path="/var/lib/kubelet/pods/1f54f71b-4cc8-4335-a50a-a34cb07e7c52/volumes" Oct 30 22:31:29.780095 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.780052 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl"] Oct 30 22:31:29.780917 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.780897 4859 cpu_manager.go:414] "containerMap: removing container" podUID="1f54f71b-4cc8-4335-a50a-a34cb07e7c52" containerName="maas-api" Oct 30 22:31:29.780978 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.780920 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f54f71b-4cc8-4335-a50a-a34cb07e7c52" containerName="maas-api" Oct 30 22:31:29.781126 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.781113 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f54f71b-4cc8-4335-a50a-a34cb07e7c52" containerName="maas-api" Oct 30 22:31:29.786318 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.786297 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:29.841324 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.841280 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl"] Oct 30 22:31:29.847614 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.847579 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/964b48c3-e385-4359-9b3f-91df3e092b08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:29.847903 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.847874 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:29.847903 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.847911 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:29.848150 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.848056 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwzm\" (UniqueName: \"kubernetes.io/projected/964b48c3-e385-4359-9b3f-91df3e092b08-kube-api-access-mdwzm\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:29.887739 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.887700 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-a0116e-serving-cert\"" Oct 30 22:31:29.949241 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.949196 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:29.949461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.949250 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:29.949461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.949361 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwzm\" (UniqueName: \"kubernetes.io/projected/964b48c3-e385-4359-9b3f-91df3e092b08-kube-api-access-mdwzm\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:29.949461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.949442 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/964b48c3-e385-4359-9b3f-91df3e092b08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:29.949799 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:31:29.949773 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-serving-cert: secret "maas-api-e2e-shared-a-w2-a0116e-serving-cert" not found Oct 30 22:31:29.950601 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:31:29.949857 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-maas-api-tls podName:964b48c3-e385-4359-9b3f-91df3e092b08 nodeName:}" failed. No retries permitted until 2026-10-30 22:31:30.449835148 +0000 UTC m=+2947.822461887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-maas-api-tls") pod "maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" (UID: "964b48c3-e385-4359-9b3f-91df3e092b08") : secret "maas-api-e2e-shared-a-w2-a0116e-serving-cert" not found Oct 30 22:31:29.950601 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.950262 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/964b48c3-e385-4359-9b3f-91df3e092b08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:29.952275 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:29.952252 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:30.006772 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:30.006737 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwzm\" (UniqueName: \"kubernetes.io/projected/964b48c3-e385-4359-9b3f-91df3e092b08-kube-api-access-mdwzm\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:30.459651 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:30.459605 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:30.462105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:30.462077 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:30.701211 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:30.701169 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:31.083794 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:31.083765 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl"] Oct 30 22:31:31.084478 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:31:31.084443 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964b48c3_e385_4359_9b3f_91df3e092b08.slice/crio-086e0f2c43ceebbe9612d8a3141dbdb7aacbf55aabae05a4702fba7ed6cdf368 WatchSource:0}: Error finding container 086e0f2c43ceebbe9612d8a3141dbdb7aacbf55aabae05a4702fba7ed6cdf368: Status 404 returned error can't find the container with id 086e0f2c43ceebbe9612d8a3141dbdb7aacbf55aabae05a4702fba7ed6cdf368 Oct 30 22:31:31.323119 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:31.323086 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" event={"ID":"964b48c3-e385-4359-9b3f-91df3e092b08","Type":"ContainerStarted","Data":"086e0f2c43ceebbe9612d8a3141dbdb7aacbf55aabae05a4702fba7ed6cdf368"} Oct 30 22:31:31.754140 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:31.754101 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc"] Oct 30 22:31:31.754352 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:31.754324 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" containerID="cri-o://842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe" gracePeriod=30 Oct 30 22:31:32.331568 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:32.331476 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" event={"ID":"964b48c3-e385-4359-9b3f-91df3e092b08","Type":"ContainerStarted","Data":"261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc"} Oct 30 22:31:32.331968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:32.331653 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:32.446848 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:32.446786 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" podStartSLOduration=2.498905106 podStartE2EDuration="3.446771452s" totalImagesPullingTime="947.866346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:32.446150321 +0000 UTC m=+2949.818777103" watchObservedRunningTime="2026-10-30 22:31:32.446771452 +0000 UTC m=+2949.819398199" Oct 30 22:31:32.993808 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:32.993771 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:31:37.036760 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.036715 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:31:37.148058 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148023 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d91468a-3e76-485e-8cac-123a4e839b86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d91468a-3e76-485e-8cac-123a4e839b86-istio-podinfo\") pod \"1d91468a-3e76-485e-8cac-123a4e839b86\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " Oct 30 22:31:37.148257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148068 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-data\") pod \"1d91468a-3e76-485e-8cac-123a4e839b86\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " Oct 30 22:31:37.148257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148158 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-certs\") pod \"1d91468a-3e76-485e-8cac-123a4e839b86\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " Oct 30 22:31:37.148257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148201 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-credential-socket\") pod \"1d91468a-3e76-485e-8cac-123a4e839b86\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " Oct 30 22:31:37.148257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148223 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-istio-token\" (UniqueName: \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-istio-token\") pod \"1d91468a-3e76-485e-8cac-123a4e839b86\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " Oct 30 22:31:37.148484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148277 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d91468a-3e76-485e-8cac-123a4e839b86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d91468a-3e76-485e-8cac-123a4e839b86-istiod-ca-cert\") pod \"1d91468a-3e76-485e-8cac-123a4e839b86\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " Oct 30 22:31:37.148484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148320 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-socket\") pod \"1d91468a-3e76-485e-8cac-123a4e839b86\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " Oct 30 22:31:37.148484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148351 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-kube-api-access-bd9jj\" (UniqueName: \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-kube-api-access-bd9jj\") pod \"1d91468a-3e76-485e-8cac-123a4e839b86\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " Oct 30 22:31:37.148484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148392 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-envoy\") pod \"1d91468a-3e76-485e-8cac-123a4e839b86\" (UID: \"1d91468a-3e76-485e-8cac-123a4e839b86\") " Oct 30 22:31:37.148484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148456 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-certs" pod "1d91468a-3e76-485e-8cac-123a4e839b86" (UID: "1d91468a-3e76-485e-8cac-123a4e839b86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:37.148752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148467 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-data" pod "1d91468a-3e76-485e-8cac-123a4e839b86" (UID: "1d91468a-3e76-485e-8cac-123a4e839b86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:37.148752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148519 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-credential-socket" pod "1d91468a-3e76-485e-8cac-123a4e839b86" (UID: "1d91468a-3e76-485e-8cac-123a4e839b86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:37.148752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148662 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-socket" pod "1d91468a-3e76-485e-8cac-123a4e839b86" (UID: "1d91468a-3e76-485e-8cac-123a4e839b86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:37.148752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148694 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d91468a-3e76-485e-8cac-123a4e839b86-istiod-ca-cert" pod "1d91468a-3e76-485e-8cac-123a4e839b86" (UID: "1d91468a-3e76-485e-8cac-123a4e839b86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:37.149000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148928 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.149000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.148986 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.149120 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.149003 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.149120 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.149015 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d91468a-3e76-485e-8cac-123a4e839b86-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.149120 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.149023 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.150537 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.150514 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d91468a-3e76-485e-8cac-123a4e839b86-istio-podinfo" pod "1d91468a-3e76-485e-8cac-123a4e839b86" (UID: "1d91468a-3e76-485e-8cac-123a4e839b86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:37.150657 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.150607 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-istio-token" pod "1d91468a-3e76-485e-8cac-123a4e839b86" (UID: "1d91468a-3e76-485e-8cac-123a4e839b86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:37.150869 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.150845 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-kube-api-access-bd9jj" pod "1d91468a-3e76-485e-8cac-123a4e839b86" (UID: "1d91468a-3e76-485e-8cac-123a4e839b86"). InnerVolumeSpecName "kube-api-access-bd9jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:37.151002 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.150925 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-envoy" pod "1d91468a-3e76-485e-8cac-123a4e839b86" (UID: "1d91468a-3e76-485e-8cac-123a4e839b86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:37.250238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.250198 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.250238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.250232 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd9jj\" (UniqueName: \"kubernetes.io/projected/1d91468a-3e76-485e-8cac-123a4e839b86-kube-api-access-bd9jj\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.250238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.250242 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d91468a-3e76-485e-8cac-123a4e839b86-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.250470 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.250254 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d91468a-3e76-485e-8cac-123a4e839b86-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.367610 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.367578 4859 generic.go:356] "Generic (PLEG): container finished" podID="1d91468a-3e76-485e-8cac-123a4e839b86" containerID="842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe" exitCode=0 Oct 30 22:31:37.367799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.367645 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" Oct 30 22:31:37.367799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.367664 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" event={"ID":"1d91468a-3e76-485e-8cac-123a4e839b86","Type":"ContainerDied","Data":"842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe"} Oct 30 22:31:37.367799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.367725 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc" event={"ID":"1d91468a-3e76-485e-8cac-123a4e839b86","Type":"ContainerDied","Data":"7ff36be87ae6fe4300166db5af769e7e9ed06b51b584a49f386ee07797aa7437"} Oct 30 22:31:37.367799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.367751 4859 scope.go:122] "RemoveContainer" containerID="842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe" Oct 30 22:31:37.383012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.382845 4859 scope.go:122] "RemoveContainer" containerID="21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274" Oct 30 22:31:37.432513 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.432482 4859 scope.go:122] "RemoveContainer" containerID="842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe" Oct 30 22:31:37.432908 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:31:37.432871 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe\": container with ID starting with 842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe not found: ID does not exist" containerID="842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe" Oct 30 22:31:37.433039 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.432911 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe"} err="failed to get container status \"842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe\": rpc error: code = NotFound desc = could not find container \"842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe\": container with ID starting with 842eba9746bfeef63983f604be95fe5af654305bff9e09dc78369d98ca48f5fe not found: ID does not exist" Oct 30 22:31:37.433039 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.432935 4859 scope.go:122] "RemoveContainer" containerID="21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274" Oct 30 22:31:37.433692 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:31:37.433438 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274\": container with ID starting with 21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274 not found: ID does not exist" containerID="21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274" Oct 30 22:31:37.433692 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.433472 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274"} err="failed to get container status \"21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274\": rpc error: code = NotFound desc = could not find container \"21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274\": container with ID starting with 21af2174c2c9f1592be884d09208cb0a88e9978860022f9c1612a7c2ff987274 not found: ID does not exist" Oct 30 22:31:37.540596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.540559 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc"] Oct 30 22:31:37.544329 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:37.544298 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b563fd-openshift-default-7797f8c88f-m5fgc"] Oct 30 22:31:38.343922 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:38.343891 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:31:38.741498 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:38.741415 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" path="/var/lib/kubelet/pods/1d91468a-3e76-485e-8cac-123a4e839b86/volumes" Oct 30 22:31:39.886107 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:31:39.886071 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:31:58.782122 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:31:58.782082 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:32:03.555548 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:03.555511 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d"] Oct 30 22:32:03.556007 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:03.555750 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" containerID="cri-o://37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923" gracePeriod=30 Oct 30 22:32:07.588162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:07.588118 4859 patch_prober.go:29] interesting pod/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:32:07.588621 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:07.588185 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:32:08.862808 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.862774 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:32:08.954622 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.954582 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e010b301-bf27-4059-ad1c-0e9e94d05684-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e010b301-bf27-4059-ad1c-0e9e94d05684-istiod-ca-cert\") pod \"e010b301-bf27-4059-ad1c-0e9e94d05684\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " Oct 30 22:32:08.954799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.954690 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-kube-api-access-bn7c5\" (UniqueName: \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-kube-api-access-bn7c5\") pod \"e010b301-bf27-4059-ad1c-0e9e94d05684\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " Oct 30 22:32:08.954799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.954738 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-token\" (UniqueName: \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-token\") pod \"e010b301-bf27-4059-ad1c-0e9e94d05684\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " Oct 30 22:32:08.954799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.954773 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-credential-socket\") pod \"e010b301-bf27-4059-ad1c-0e9e94d05684\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " Oct 30 22:32:08.954938 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.954799 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-data\") pod \"e010b301-bf27-4059-ad1c-0e9e94d05684\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " Oct 30 22:32:08.954938 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.954903 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-podinfo\") pod \"e010b301-bf27-4059-ad1c-0e9e94d05684\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " Oct 30 22:32:08.955069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.954965 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-envoy\") pod \"e010b301-bf27-4059-ad1c-0e9e94d05684\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " Oct 30 22:32:08.955069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955017 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-certs\") pod \"e010b301-bf27-4059-ad1c-0e9e94d05684\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " Oct 30 22:32:08.955069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955027 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e010b301-bf27-4059-ad1c-0e9e94d05684-istiod-ca-cert" pod "e010b301-bf27-4059-ad1c-0e9e94d05684" (UID: "e010b301-bf27-4059-ad1c-0e9e94d05684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:08.955234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955169 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-credential-socket" pod "e010b301-bf27-4059-ad1c-0e9e94d05684" (UID: "e010b301-bf27-4059-ad1c-0e9e94d05684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:08.955234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955208 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-data" pod "e010b301-bf27-4059-ad1c-0e9e94d05684" (UID: "e010b301-bf27-4059-ad1c-0e9e94d05684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:08.955341 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955114 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-socket\") pod \"e010b301-bf27-4059-ad1c-0e9e94d05684\" (UID: \"e010b301-bf27-4059-ad1c-0e9e94d05684\") " Oct 30 22:32:08.955396 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955317 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-socket" pod "e010b301-bf27-4059-ad1c-0e9e94d05684" (UID: "e010b301-bf27-4059-ad1c-0e9e94d05684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:08.955445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955386 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-certs" pod "e010b301-bf27-4059-ad1c-0e9e94d05684" (UID: "e010b301-bf27-4059-ad1c-0e9e94d05684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:08.955741 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955715 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:08.955741 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955743 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:08.955936 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955759 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e010b301-bf27-4059-ad1c-0e9e94d05684-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:08.955936 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955774 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:08.955936 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.955787 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:08.957318 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.957293 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-token" pod "e010b301-bf27-4059-ad1c-0e9e94d05684" (UID: "e010b301-bf27-4059-ad1c-0e9e94d05684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:08.957486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.957459 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-envoy" pod "e010b301-bf27-4059-ad1c-0e9e94d05684" (UID: "e010b301-bf27-4059-ad1c-0e9e94d05684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:08.957542 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.957517 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-kube-api-access-bn7c5" pod "e010b301-bf27-4059-ad1c-0e9e94d05684" (UID: "e010b301-bf27-4059-ad1c-0e9e94d05684"). InnerVolumeSpecName "kube-api-access-bn7c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:08.957631 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:08.957603 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-podinfo" pod "e010b301-bf27-4059-ad1c-0e9e94d05684" (UID: "e010b301-bf27-4059-ad1c-0e9e94d05684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:09.057191 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.057148 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn7c5\" (UniqueName: \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-kube-api-access-bn7c5\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:09.057191 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.057188 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:09.057389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.057202 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:09.057389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.057215 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e010b301-bf27-4059-ad1c-0e9e94d05684-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:09.655968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.655410 4859 generic.go:356] "Generic (PLEG): container finished" podID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerID="efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98" exitCode=0 Oct 30 22:32:09.655968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.655550 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" event={"ID":"e3efccd2-b4f7-47ad-ac24-326f5246aac6","Type":"ContainerDied","Data":"efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98"} Oct 30 22:32:09.656278 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.656086 4859 scope.go:122] "RemoveContainer" containerID="efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98" Oct 30 22:32:09.657778 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.657753 4859 generic.go:356] "Generic (PLEG): container finished" podID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerID="37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923" exitCode=0 Oct 30 22:32:09.657905 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.657804 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" Oct 30 22:32:09.657905 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.657827 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" event={"ID":"e010b301-bf27-4059-ad1c-0e9e94d05684","Type":"ContainerDied","Data":"37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923"} Oct 30 22:32:09.657905 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.657890 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d" event={"ID":"e010b301-bf27-4059-ad1c-0e9e94d05684","Type":"ContainerDied","Data":"647e46b2e4f7c6ba34bac927fa5ba999a0e9738919730bfae451aa4d0fadb011"} Oct 30 22:32:09.658077 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.657890 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:09.658077 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.657922 4859 scope.go:122] "RemoveContainer" containerID="37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923" Oct 30 22:32:09.658077 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.658009 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:09.658077 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.658058 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:09.677190 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.677160 4859 scope.go:122] "RemoveContainer" containerID="f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe" Oct 30 22:32:09.720046 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.719998 4859 scope.go:122] "RemoveContainer" containerID="37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923" Oct 30 22:32:09.721831 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:32:09.720815 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923\": container with ID starting with 37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923 not found: ID does not exist" containerID="37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923" Oct 30 22:32:09.721831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.720846 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923"} err="failed to get container status \"37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923\": rpc error: code = NotFound desc = could not find container \"37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923\": container with ID starting with 37f2f4aeae755b363cffe88de69559cfdf5886bf2dceeaffa97ab24b48f6e923 not found: ID does not exist" Oct 30 22:32:09.721831 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.720864 4859 scope.go:122] "RemoveContainer" containerID="f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe" Oct 30 22:32:09.722086 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:32:09.722059 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe\": container with ID starting with f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe not found: ID does not exist" containerID="f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe" Oct 30 22:32:09.722125 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.722096 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe"} err="failed to get container status \"f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe\": rpc error: code = NotFound desc = could not find container \"f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe\": container with ID starting with f62bc7666f276edef6fc73ab3d8fdffb2dbb9961243b597800132d6a06796ffe not found: ID does not exist" Oct 30 22:32:09.893052 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:32:09.888922 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:32:09.893550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.893404 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d"] Oct 30 22:32:09.896916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:09.896888 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-42e8a1-openshift-default-7987cb45c7-zw97d"] Oct 30 22:32:10.713299 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:10.713260 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" event={"ID":"e3efccd2-b4f7-47ad-ac24-326f5246aac6","Type":"ContainerStarted","Data":"ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13"} Oct 30 22:32:10.740875 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:10.740840 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" path="/var/lib/kubelet/pods/e010b301-bf27-4059-ad1c-0e9e94d05684/volumes" Oct 30 22:32:10.802669 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:10.802632 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:32:10.803791 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:10.803757 4859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:32:10.803961 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:10.803834 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:32:11.692091 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:11.692051 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:32:11.802978 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:11.802921 4859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:32:11.803172 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:11.803020 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:32:12.802393 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:12.802353 4859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:32:12.803056 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:12.803011 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:32:13.802448 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:13.802408 4859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:32:13.802868 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:13.802481 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:32:14.802516 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:14.802480 4859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:32:14.802973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:14.802548 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:32:15.817580 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:15.817526 4859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:32:15.818123 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:15.817638 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:32:16.806777 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:16.806736 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:32:16.807743 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:16.807720 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:32:23.301420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:23.301387 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:32:23.311165 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:23.311119 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:32:23.323408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:23.323383 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:32:23.331367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:23.331340 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:32:30.631982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.631048 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655"] Oct 30 22:32:30.631982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.631926 4859 cpu_manager.go:414] "containerMap: removing container" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" Oct 30 22:32:30.631982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.631969 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" Oct 30 22:32:30.632703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.631997 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" Oct 30 22:32:30.632703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.632007 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" Oct 30 22:32:30.632703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.632198 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" Oct 30 22:32:30.632703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.632216 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" Oct 30 22:32:30.632703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.632228 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" Oct 30 22:32:30.639963 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.636047 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.687017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.685855 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-socket\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.687017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.686003 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-data\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.687017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.686073 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-envoy\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.687017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.686169 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb061db-828f-4cc8-aa1f-116685012e01-istio-podinfo\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.687017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.686214 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv986\" (UniqueName: \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-kube-api-access-mv986\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.687017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.686291 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-credential-socket\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.687017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.686345 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb061db-828f-4cc8-aa1f-116685012e01-istiod-ca-cert\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.687017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.686428 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-istio-token\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.687017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.686483 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-certs\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.688004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.687984 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6c788d-openshift-default-dockercfg-zpp47\"" Oct 30 22:32:30.744596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.744564 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655"] Oct 30 22:32:30.796782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.787660 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-envoy\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.796782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.787733 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb061db-828f-4cc8-aa1f-116685012e01-istio-podinfo\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.796782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.787792 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv986\" (UniqueName: \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-kube-api-access-mv986\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.796782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.787843 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-credential-socket\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.796782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.787883 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb061db-828f-4cc8-aa1f-116685012e01-istiod-ca-cert\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.796782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.787983 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-istio-token\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.796782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.788073 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-certs\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.796782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.788132 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-socket\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.796782 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.788201 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-data\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.814254 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.812937 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-certs\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.814254 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.813227 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-socket\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.814254 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.813673 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-data\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.814576 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.814497 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-credential-socket\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.820031 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.815145 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb061db-828f-4cc8-aa1f-116685012e01-istiod-ca-cert\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.820276 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.817199 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-envoy\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.820276 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.817634 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb061db-828f-4cc8-aa1f-116685012e01-istio-podinfo\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.868394 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.868360 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-istio-token\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.869129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.869107 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv986\" (UniqueName: \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-kube-api-access-mv986\") pod \"e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:30.952762 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:30.952668 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:31.163054 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:31.163014 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:32:31.384916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:31.384882 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655"] Oct 30 22:32:31.385064 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:32:31.384916 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb061db_828f_4cc8_aa1f_116685012e01.slice/crio-c9516bbeb69022c479d322c8904622c0a9f1c1ff6d45c02188fc407d18437efc WatchSource:0}: Error finding container c9516bbeb69022c479d322c8904622c0a9f1c1ff6d45c02188fc407d18437efc: Status 404 returned error can't find the container with id c9516bbeb69022c479d322c8904622c0a9f1c1ff6d45c02188fc407d18437efc Oct 30 22:32:31.387624 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:31.387586 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:31.387709 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:31.387666 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:31.387747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:31.387712 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:31.962222 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:31.962183 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" event={"ID":"fdb061db-828f-4cc8-aa1f-116685012e01","Type":"ContainerStarted","Data":"9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f"} Oct 30 22:32:31.962629 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:31.962246 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" event={"ID":"fdb061db-828f-4cc8-aa1f-116685012e01","Type":"ContainerStarted","Data":"c9516bbeb69022c479d322c8904622c0a9f1c1ff6d45c02188fc407d18437efc"} Oct 30 22:32:32.081172 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:32.081116 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" podStartSLOduration=2.081102013 podStartE2EDuration="2.081102013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:32.078752889 +0000 UTC m=+3009.451379637" watchObservedRunningTime="2026-10-30 22:32:32.081102013 +0000 UTC m=+3009.453728760" Oct 30 22:32:32.953151 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:32.953118 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:32.958496 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:32.958469 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:32.968867 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:32.968844 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:32.969578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:32.969560 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:32:38.924975 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:38.924613 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms"] Oct 30 22:32:38.927244 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:38.927220 4859 cpu_manager.go:414] "containerMap: removing container" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" Oct 30 22:32:38.927403 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:38.927390 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" Oct 30 22:32:38.927494 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:38.927485 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" Oct 30 22:32:38.927563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:38.927553 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e010b301-bf27-4059-ad1c-0e9e94d05684" containerName="istio-proxy" Oct 30 22:32:38.927816 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:38.927803 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d91468a-3e76-485e-8cac-123a4e839b86" containerName="istio-proxy" Oct 30 22:32:38.933531 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:38.933514 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:38.990062 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:38.990025 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms"] Oct 30 22:32:39.027746 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.027709 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-socket\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.027923 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.027763 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-certs\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.027923 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.027893 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-credential-socket\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.028049 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.027933 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-envoy\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.028049 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.028031 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mqwj\" (UniqueName: \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-kube-api-access-7mqwj\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.028137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.028087 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istiod-ca-cert\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.028214 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.028198 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-data\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.028322 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.028299 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-podinfo\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.028412 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.028392 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-token\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.033684 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.033666 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-b1eb3f-openshift-default-dockercfg-gm8nc\"" Oct 30 22:32:39.129709 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.129676 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-data\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.129889 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.129727 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-podinfo\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.129889 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.129772 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-token\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.129889 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.129803 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-socket\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.129889 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.129831 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-certs\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.129889 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.129887 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-credential-socket\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.130195 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.129915 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-envoy\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.130195 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.129939 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mqwj\" (UniqueName: \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-kube-api-access-7mqwj\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.130195 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.129993 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istiod-ca-cert\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.130358 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.130199 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-data\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.130418 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.130356 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-socket\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.130418 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.130404 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-certs\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.130530 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.130452 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-credential-socket\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.130735 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.130715 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istiod-ca-cert\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.132220 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.132197 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-envoy\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.132358 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.132341 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-podinfo\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.186774 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.186708 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mqwj\" (UniqueName: \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-kube-api-access-7mqwj\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.186774 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.186736 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-token\") pod \"e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.253332 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.253285 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:39.689976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.689924 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms"] Oct 30 22:32:39.691097 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:32:39.691070 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1491e9_ac51_4e0b_828a_fe8984ff6282.slice/crio-499b894763e73933e9abe9c67b23ca268ada15381e2bacb198948dcb0f49a67d WatchSource:0}: Error finding container 499b894763e73933e9abe9c67b23ca268ada15381e2bacb198948dcb0f49a67d: Status 404 returned error can't find the container with id 499b894763e73933e9abe9c67b23ca268ada15381e2bacb198948dcb0f49a67d Oct 30 22:32:39.693570 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.693530 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:39.693671 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.693603 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:39.693671 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:39.693631 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:39.901974 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:32:39.901350 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:32:40.028845 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:40.028744 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" event={"ID":"2c1491e9-ac51-4e0b-828a-fe8984ff6282","Type":"ContainerStarted","Data":"732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa"} Oct 30 22:32:40.028845 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:40.028816 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" event={"ID":"2c1491e9-ac51-4e0b-828a-fe8984ff6282","Type":"ContainerStarted","Data":"499b894763e73933e9abe9c67b23ca268ada15381e2bacb198948dcb0f49a67d"} Oct 30 22:32:40.146306 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:40.146252 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" podStartSLOduration=2.146238362 podStartE2EDuration="2.146238362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:40.144638536 +0000 UTC m=+3017.517265282" watchObservedRunningTime="2026-10-30 22:32:40.146238362 +0000 UTC m=+3017.518865109" Oct 30 22:32:40.267982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:40.257796 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:40.293028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:40.292891 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:41.035660 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:41.035625 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:32:41.036732 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:32:41.036710 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:33:09.877599 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:33:09.877559 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:33:12.285508 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:12.285467 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:33:37.974026 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:37.973989 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:33:38.891851 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.891818 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx"] Oct 30 22:33:38.900231 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.900195 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:38.953041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.953001 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-e90e35-openshift-default-dockercfg-4j6lz\"" Oct 30 22:33:38.979638 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.979607 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99dtb\" (UniqueName: \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-kube-api-access-99dtb\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:38.980012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.979649 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:38.980012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.979666 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:38.980012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.979794 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:38.980012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.979854 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:38.980012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.979881 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:38.980012 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.979938 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:38.980230 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.980035 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:38.980230 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:38.980096 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.006496 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.006460 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx"] Oct 30 22:33:39.080769 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.080729 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.080928 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.080779 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.080928 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.080804 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.080928 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.080822 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.080928 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.080877 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.081167 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.080936 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.081167 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.081007 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99dtb\" (UniqueName: \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-kube-api-access-99dtb\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.081167 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.081050 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.081167 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.081073 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.081517 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.081492 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.081689 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.081524 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.081786 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.081575 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.081786 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.081619 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.081872 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.081802 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.083183 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.083160 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.083441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.083423 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.138111 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.138090 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.138333 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.138310 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99dtb\" (UniqueName: \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-kube-api-access-99dtb\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.214456 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.214386 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:39.694044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.690872 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx"] Oct 30 22:33:39.696708 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.696672 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:33:39.696841 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.696746 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:33:39.696841 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:39.696787 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:33:39.903742 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:33:39.903709 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:33:40.591966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:40.591558 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" event={"ID":"f5f748cb-e716-42f1-ba4e-cfd4aca65d04","Type":"ContainerStarted","Data":"0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed"} Oct 30 22:33:40.591966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:40.591660 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" event={"ID":"f5f748cb-e716-42f1-ba4e-cfd4aca65d04","Type":"ContainerStarted","Data":"b94cca08eb1c98ff16d6316dcd61115933f0c2afca2b23dc3672d07acb313923"} Oct 30 22:33:40.813973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:40.813879 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" podStartSLOduration=2.813859875 podStartE2EDuration="2.813859875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:40.76080648 +0000 UTC m=+3078.133433236" watchObservedRunningTime="2026-10-30 22:33:40.813859875 +0000 UTC m=+3078.186486622" Oct 30 22:33:41.217966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:41.217329 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:41.220966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:41.219826 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:33:41.220966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:41.219881 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 22:33:42.215969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:42.215012 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:33:42.215969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:42.215069 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 22:33:43.215968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:43.215351 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:33:43.215968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:43.215415 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 22:33:44.215968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:44.215352 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:33:44.215968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:44.215415 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 22:33:45.215609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:45.215572 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:33:45.215785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:45.215629 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 22:33:46.215818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:46.215769 4859 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:33:46.216247 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:46.215841 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 22:33:47.220342 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:47.220310 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:47.220792 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:47.220777 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:47.221562 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:47.221544 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:33:48.421830 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:48.421797 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": read tcp 10.132.0.2:42580->10.132.0.153:15021: read: connection reset by peer" start-of-body= Oct 30 22:33:48.422531 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:48.421861 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": read tcp 10.132.0.2:42580->10.132.0.153:15021: read: connection reset by peer" Oct 30 22:33:48.673373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:48.673290 4859 generic.go:356] "Generic (PLEG): container finished" podID="fdb061db-828f-4cc8-aa1f-116685012e01" containerID="9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f" exitCode=0 Oct 30 22:33:48.673541 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:48.673364 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" event={"ID":"fdb061db-828f-4cc8-aa1f-116685012e01","Type":"ContainerDied","Data":"9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f"} Oct 30 22:33:48.673873 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:48.673854 4859 scope.go:122] "RemoveContainer" containerID="9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f" Oct 30 22:33:48.675168 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:48.675139 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:33:48.675224 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:48.675210 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:33:48.675263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:48.675239 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:33:49.691222 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:49.691183 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" event={"ID":"fdb061db-828f-4cc8-aa1f-116685012e01","Type":"ContainerStarted","Data":"30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92"} Oct 30 22:33:49.953422 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:49.953344 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:33:49.953563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:49.953479 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:33:49.953563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:49.953524 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:33:50.953295 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:50.953255 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:33:50.953689 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:50.953312 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:33:51.953401 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:51.953362 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:33:51.953811 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:51.953423 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:33:52.953773 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:52.953735 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:33:52.954190 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:52.953790 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:33:53.953894 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:53.953857 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:33:53.954387 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:53.953914 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:33:54.957362 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:54.957331 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:33:54.957842 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:54.957819 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:33:54.958544 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:33:54.958524 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:34:09.898265 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:34:09.898184 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:34:16.876194 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:16.876140 4859 generic.go:356] "Generic (PLEG): container finished" podID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerID="732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa" exitCode=0 Oct 30 22:34:16.876559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:16.876217 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" event={"ID":"2c1491e9-ac51-4e0b-828a-fe8984ff6282","Type":"ContainerDied","Data":"732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa"} Oct 30 22:34:16.876603 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:16.876572 4859 scope.go:122] "RemoveContainer" containerID="732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa" Oct 30 22:34:16.877912 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:16.877883 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:34:16.877994 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:16.877965 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:34:16.878038 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:16.878001 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:34:17.956779 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:17.956739 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" event={"ID":"2c1491e9-ac51-4e0b-828a-fe8984ff6282","Type":"ContainerStarted","Data":"deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97"} Oct 30 22:34:18.254204 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:18.254130 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:34:18.254464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:18.254444 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 22:34:18.254522 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:18.254496 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 22:34:19.254230 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:19.254201 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 22:34:19.254597 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:19.254248 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 22:34:20.253808 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:20.253775 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 22:34:20.254096 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:20.253826 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 22:34:21.254119 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.254087 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 22:34:21.254496 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.254138 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 22:34:21.300315 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.300274 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs"] Oct 30 22:34:21.305377 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.305357 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.348729 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.348700 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs"] Oct 30 22:34:21.357128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.357107 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-89cf28-serving-cert\"" Oct 30 22:34:21.401441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.401415 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmsz\" (UniqueName: \"kubernetes.io/projected/da81d643-efd0-4edb-a187-01a2e1a63641-kube-api-access-dcmsz\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.401606 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.401475 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.401606 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.401532 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da81d643-efd0-4edb-a187-01a2e1a63641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.401606 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.401550 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.503088 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.503052 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da81d643-efd0-4edb-a187-01a2e1a63641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.503276 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.503095 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.503276 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.503254 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmsz\" (UniqueName: \"kubernetes.io/projected/da81d643-efd0-4edb-a187-01a2e1a63641-kube-api-access-dcmsz\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.503276 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:34:21.503263 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-serving-cert: secret "maas-api-e2e-shared-a-w3-89cf28-serving-cert" not found Oct 30 22:34:21.503444 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:34:21.503320 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-maas-api-tls podName:da81d643-efd0-4edb-a187-01a2e1a63641 nodeName:}" failed. No retries permitted until 2026-10-30 22:34:22.003306032 +0000 UTC m=+3119.375932756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-maas-api-tls") pod "maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" (UID: "da81d643-efd0-4edb-a187-01a2e1a63641") : secret "maas-api-e2e-shared-a-w3-89cf28-serving-cert" not found Oct 30 22:34:21.503444 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.503349 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.503668 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.503649 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da81d643-efd0-4edb-a187-01a2e1a63641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.505705 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.505652 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:21.583018 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:21.573853 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmsz\" (UniqueName: \"kubernetes.io/projected/da81d643-efd0-4edb-a187-01a2e1a63641-kube-api-access-dcmsz\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:22.011533 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:22.011495 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:22.013887 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:22.013859 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:22.219323 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:22.219290 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:22.254748 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:22.254552 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 22:34:22.254748 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:22.254615 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 22:34:22.624429 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:34:22.624397 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda81d643_efd0_4edb_a187_01a2e1a63641.slice/crio-c8a48e260c0fbc92aae093a40b5a03a659026e5685014ccbcaba4ae4ba5fc1a0 WatchSource:0}: Error finding container c8a48e260c0fbc92aae093a40b5a03a659026e5685014ccbcaba4ae4ba5fc1a0: Status 404 returned error can't find the container with id c8a48e260c0fbc92aae093a40b5a03a659026e5685014ccbcaba4ae4ba5fc1a0 Oct 30 22:34:22.624918 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:22.624895 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs"] Oct 30 22:34:22.626174 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:22.626157 4859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:34:23.016197 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:23.016120 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" event={"ID":"da81d643-efd0-4edb-a187-01a2e1a63641","Type":"ContainerStarted","Data":"c8a48e260c0fbc92aae093a40b5a03a659026e5685014ccbcaba4ae4ba5fc1a0"} Oct 30 22:34:23.258548 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:23.258519 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:34:23.258904 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:23.258848 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:34:23.259592 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:23.259575 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:34:24.024441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:24.024400 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" event={"ID":"da81d643-efd0-4edb-a187-01a2e1a63641","Type":"ContainerStarted","Data":"6213dde8196dfd9e49d4a836c2100b46a1ad90e8d18ac6d3e8e904f2caf128a2"} Oct 30 22:34:24.024616 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:24.024495 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:24.143449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:24.143405 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" podStartSLOduration=1.97198143 podStartE2EDuration="3.143391519s" totalImagesPullingTime="1.171410089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:24.141419852 +0000 UTC m=+3121.514046608" watchObservedRunningTime="2026-10-30 22:34:24.143391519 +0000 UTC m=+3121.516018304" Oct 30 22:34:30.040102 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:30.040070 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:34:33.767714 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:34:33.767660 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:34:39.878989 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:34:39.878933 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:35:09.879876 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:35:09.879831 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:35:13.811992 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:13.811955 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:35:16.199110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.199077 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk"] Oct 30 22:35:16.203311 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.203287 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.308664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.308630 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-9f7654-serving-cert\"" Oct 30 22:35:16.312175 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.312146 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk"] Oct 30 22:35:16.356255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.356220 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2hhn\" (UniqueName: \"kubernetes.io/projected/e68cb285-6e5e-4918-9962-fd8cc8e8680f-kube-api-access-z2hhn\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.356418 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.356271 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68cb285-6e5e-4918-9962-fd8cc8e8680f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.356464 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.356429 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.356570 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.356553 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.483982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.466709 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.483982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.466883 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.483982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.467011 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2hhn\" (UniqueName: \"kubernetes.io/projected/e68cb285-6e5e-4918-9962-fd8cc8e8680f-kube-api-access-z2hhn\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.483982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.467046 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68cb285-6e5e-4918-9962-fd8cc8e8680f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.483982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.467995 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68cb285-6e5e-4918-9962-fd8cc8e8680f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.483982 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:35:16.468104 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-serving-cert: secret "maas-api-e2e-shared-b-w1-9f7654-serving-cert" not found Oct 30 22:35:16.483982 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:35:16.468167 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-maas-api-tls podName:e68cb285-6e5e-4918-9962-fd8cc8e8680f nodeName:}" failed. No retries permitted until 2026-10-30 22:35:16.968146986 +0000 UTC m=+3174.340773751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-maas-api-tls") pod "maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" (UID: "e68cb285-6e5e-4918-9962-fd8cc8e8680f") : secret "maas-api-e2e-shared-b-w1-9f7654-serving-cert" not found Oct 30 22:35:16.490972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.490767 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.528429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.528397 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2hhn\" (UniqueName: \"kubernetes.io/projected/e68cb285-6e5e-4918-9962-fd8cc8e8680f-kube-api-access-z2hhn\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.972836 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.972797 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:16.975881 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:16.975850 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:17.117676 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:17.117642 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:17.502335 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:17.502306 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk"] Oct 30 22:35:17.503708 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:35:17.503684 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68cb285_6e5e_4918_9962_fd8cc8e8680f.slice/crio-c7be44911037d6b74d73640267fd1b2274ac483c8d04a86516ae541b8ea099d6 WatchSource:0}: Error finding container c7be44911037d6b74d73640267fd1b2274ac483c8d04a86516ae541b8ea099d6: Status 404 returned error can't find the container with id c7be44911037d6b74d73640267fd1b2274ac483c8d04a86516ae541b8ea099d6 Oct 30 22:35:18.449244 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:18.448084 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" event={"ID":"e68cb285-6e5e-4918-9962-fd8cc8e8680f","Type":"ContainerStarted","Data":"c7be44911037d6b74d73640267fd1b2274ac483c8d04a86516ae541b8ea099d6"} Oct 30 22:35:19.456646 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:19.456609 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" event={"ID":"e68cb285-6e5e-4918-9962-fd8cc8e8680f","Type":"ContainerStarted","Data":"39fdd0e75b9a2861e1a196a02e44ee683ba3237627172740de845fc9cfc69bc3"} Oct 30 22:35:19.457134 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:19.456669 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:19.572539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:19.572472 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" podStartSLOduration=2.69853887 podStartE2EDuration="3.572452564s" totalImagesPullingTime="873.913694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:19.570040322 +0000 UTC m=+3176.942667074" watchObservedRunningTime="2026-10-30 22:35:19.572452564 +0000 UTC m=+3176.945079312" Oct 30 22:35:25.471753 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:25.471726 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:35:39.880968 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:35:39.880853 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:35:46.418201 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.418168 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq"] Oct 30 22:35:46.422401 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.422384 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.470286 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.470245 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1f86d0-dacc-4d24-8425-d8a581152da8-openshift-service-ca\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.470428 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.470367 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742tj\" (UniqueName: \"kubernetes.io/projected/4d1f86d0-dacc-4d24-8425-d8a581152da8-kube-api-access-742tj\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.470497 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.470425 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-metrics-certs\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.470684 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.470664 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-maas-api-tls\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.523940 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.523908 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-9651e9-serving-cert\"" Oct 30 22:35:46.527466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.527441 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq"] Oct 30 22:35:46.571825 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.571789 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-maas-api-tls\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.572034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.571860 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1f86d0-dacc-4d24-8425-d8a581152da8-openshift-service-ca\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.572034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.571905 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742tj\" (UniqueName: \"kubernetes.io/projected/4d1f86d0-dacc-4d24-8425-d8a581152da8-kube-api-access-742tj\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.572034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.571930 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-metrics-certs\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.572406 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:35:46.572383 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-serving-cert: secret "maas-api-e2e-explicit-9651e9-serving-cert" not found Oct 30 22:35:46.572505 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:35:46.572468 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-maas-api-tls podName:4d1f86d0-dacc-4d24-8425-d8a581152da8 nodeName:}" failed. No retries permitted until 2026-10-30 22:35:47.072447137 +0000 UTC m=+3204.445073877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-maas-api-tls") pod "maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" (UID: "4d1f86d0-dacc-4d24-8425-d8a581152da8") : secret "maas-api-e2e-explicit-9651e9-serving-cert" not found Oct 30 22:35:46.573008 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.572986 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1f86d0-dacc-4d24-8425-d8a581152da8-openshift-service-ca\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.574736 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.574711 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-metrics-certs\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:46.634701 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:46.634666 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-742tj\" (UniqueName: \"kubernetes.io/projected/4d1f86d0-dacc-4d24-8425-d8a581152da8-kube-api-access-742tj\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:47.096210 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:47.096177 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-maas-api-tls\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:47.098691 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:47.098663 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-maas-api-tls\") pod \"maas-api-e2e-explicit-9651e9-59dc58785c-x49vq\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:47.335862 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:47.335822 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:47.713688 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:47.713656 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq"] Oct 30 22:35:47.714974 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:35:47.714925 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1f86d0_dacc_4d24_8425_d8a581152da8.slice/crio-7e4796d2aea35fa6774ea240bd18c252e64ac673d8d7a9ca149e3053e1a12fa9 WatchSource:0}: Error finding container 7e4796d2aea35fa6774ea240bd18c252e64ac673d8d7a9ca149e3053e1a12fa9: Status 404 returned error can't find the container with id 7e4796d2aea35fa6774ea240bd18c252e64ac673d8d7a9ca149e3053e1a12fa9 Oct 30 22:35:48.660143 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:48.660106 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" event={"ID":"4d1f86d0-dacc-4d24-8425-d8a581152da8","Type":"ContainerStarted","Data":"7e4796d2aea35fa6774ea240bd18c252e64ac673d8d7a9ca149e3053e1a12fa9"} Oct 30 22:35:49.674418 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:49.674381 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" event={"ID":"4d1f86d0-dacc-4d24-8425-d8a581152da8","Type":"ContainerStarted","Data":"fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83"} Oct 30 22:35:49.674892 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:49.674497 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:35:49.790150 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:49.790087 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" podStartSLOduration=2.822458205 podStartE2EDuration="3.790073264s" totalImagesPullingTime="967.615059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:49.788675008 +0000 UTC m=+3207.161301757" watchObservedRunningTime="2026-10-30 22:35:49.790073264 +0000 UTC m=+3207.162700010" Oct 30 22:35:55.732146 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:35:55.732112 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:36:03.211687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:36:03.211650 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:36:09.877628 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:36:09.877588 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:36:39.877643 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:36:39.877605 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:37:09.880322 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:37:09.879020 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:37:15.501976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:37:15.501920 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:37:23.352687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:37:23.352659 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:37:23.361035 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:37:23.361010 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:37:23.374703 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:37:23.374682 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:37:23.382366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:37:23.382346 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:37:39.895817 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:37:39.895768 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:38:06.843902 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:06.843864 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:38:08.479379 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:08.479344 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp"] Oct 30 22:38:08.479784 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:08.479623 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" containerID="cri-o://5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1" gracePeriod=30 Oct 30 22:38:08.526990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:08.526959 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq"] Oct 30 22:38:08.527235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:08.527210 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerName="istio-proxy" containerID="cri-o://3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb" gracePeriod=30 Oct 30 22:38:09.878311 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:38:09.878270 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:38:13.265121 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.265087 4859 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:38:13.265503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.265155 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:38:13.797702 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.797629 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:38:13.833830 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.833808 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 22:38:13.838374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.838347 4859 generic.go:356] "Generic (PLEG): container finished" podID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerID="5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1" exitCode=0 Oct 30 22:38:13.838477 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.838381 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" event={"ID":"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b","Type":"ContainerDied","Data":"5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1"} Oct 30 22:38:13.838477 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.838433 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" event={"ID":"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b","Type":"ContainerDied","Data":"458aabc28b63c70d8adda78790f0ffe34c10d6a4dfdd92b545fb37dcf751555b"} Oct 30 22:38:13.838477 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.838454 4859 scope.go:122] "RemoveContainer" containerID="5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1" Oct 30 22:38:13.838477 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.838391 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp" Oct 30 22:38:13.839805 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.839786 4859 generic.go:356] "Generic (PLEG): container finished" podID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerID="3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb" exitCode=0 Oct 30 22:38:13.839917 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.839824 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" Oct 30 22:38:13.839917 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.839835 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" event={"ID":"b67abee5-46f6-464d-9a37-8afaf3ebdd5b","Type":"ContainerDied","Data":"3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb"} Oct 30 22:38:13.839917 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.839884 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq" event={"ID":"b67abee5-46f6-464d-9a37-8afaf3ebdd5b","Type":"ContainerDied","Data":"9e4ed61c13701d52e59d1e45a7a94ef0a1995825f9fdf0321cc71c7b54ac7d75"} Oct 30 22:38:13.848115 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.848095 4859 scope.go:122] "RemoveContainer" containerID="5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1" Oct 30 22:38:13.848417 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:38:13.848395 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1\": container with ID starting with 5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1 not found: ID does not exist" containerID="5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1" Oct 30 22:38:13.848465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.848429 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1"} err="failed to get container status \"5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1\": rpc error: code = NotFound desc = could not find container \"5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1\": container with ID starting with 5c5895c56a1786ddf8d782506eb708331735c3c020f718139ac2e68cbcee7ec1 not found: ID does not exist" Oct 30 22:38:13.848465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.848452 4859 scope.go:122] "RemoveContainer" containerID="3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb" Oct 30 22:38:13.857977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.857958 4859 scope.go:122] "RemoveContainer" containerID="3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb" Oct 30 22:38:13.858252 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:38:13.858233 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb\": container with ID starting with 3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb not found: ID does not exist" containerID="3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb" Oct 30 22:38:13.858327 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.858258 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb"} err="failed to get container status \"3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb\": rpc error: code = NotFound desc = could not find container \"3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb\": container with ID starting with 3b0ddacfc1f67961f021b6fee3cf38074a3c004c0cd1cd9ce0b76d899accfefb not found: ID does not exist" Oct 30 22:38:13.922990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.922958 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-podinfo\") pod \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " Oct 30 22:38:13.923180 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923010 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-token\" (UniqueName: \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-token\") pod \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " Oct 30 22:38:13.923180 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923063 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-data\") pod \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " Oct 30 22:38:13.923180 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923101 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-data\") pod \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " Oct 30 22:38:13.923180 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923177 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-socket\") pod \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " Oct 30 22:38:13.923409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923219 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-kube-api-access-mmv8t\" (UniqueName: \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-kube-api-access-mmv8t\") pod \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " Oct 30 22:38:13.923409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923271 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-token\" (UniqueName: \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-token\") pod \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " Oct 30 22:38:13.923409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923353 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-socket\") pod \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " Oct 30 22:38:13.923409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923386 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-kube-api-access-vgxjx\" (UniqueName: \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-kube-api-access-vgxjx\") pod \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " Oct 30 22:38:13.923628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923423 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-certs\") pod \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " Oct 30 22:38:13.923628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923405 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-data" pod "b67abee5-46f6-464d-9a37-8afaf3ebdd5b" (UID: "b67abee5-46f6-464d-9a37-8afaf3ebdd5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:13.923628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923489 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istiod-ca-cert\") pod \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " Oct 30 22:38:13.923628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923506 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-socket" pod "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" (UID: "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:13.923628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923532 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-credential-socket\") pod \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " Oct 30 22:38:13.923628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923562 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-envoy\") pod \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " Oct 30 22:38:13.923628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923607 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-certs\") pod \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " Oct 30 22:38:13.924010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923637 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-credential-socket\") pod \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " Oct 30 22:38:13.924010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923695 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istiod-ca-cert\") pod \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " Oct 30 22:38:13.924010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923721 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-envoy\") pod \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\" (UID: \"b147f1b4-2219-4fcf-977b-e6b3b72b7c7b\") " Oct 30 22:38:13.924010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923771 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-podinfo\") pod \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\" (UID: \"b67abee5-46f6-464d-9a37-8afaf3ebdd5b\") " Oct 30 22:38:13.924243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923637 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-data" pod "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" (UID: "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:13.924243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923662 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-socket" pod "b67abee5-46f6-464d-9a37-8afaf3ebdd5b" (UID: "b67abee5-46f6-464d-9a37-8afaf3ebdd5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:13.924243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.923960 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-certs" pod "b67abee5-46f6-464d-9a37-8afaf3ebdd5b" (UID: "b67abee5-46f6-464d-9a37-8afaf3ebdd5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:13.924243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924126 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-credential-socket" pod "b67abee5-46f6-464d-9a37-8afaf3ebdd5b" (UID: "b67abee5-46f6-464d-9a37-8afaf3ebdd5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:13.924740 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924320 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-credential-socket" pod "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" (UID: "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:13.924740 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924511 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istiod-ca-cert" pod "b67abee5-46f6-464d-9a37-8afaf3ebdd5b" (UID: "b67abee5-46f6-464d-9a37-8afaf3ebdd5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:13.924740 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924583 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istiod-ca-cert" pod "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" (UID: "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:13.924740 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924714 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.924740 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924720 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-certs" pod "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" (UID: "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:13.924740 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924737 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.925185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924752 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.925185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924766 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.925185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924779 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.925185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924792 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.925185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924805 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.925185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924819 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.925185 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.924833 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.926216 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.926189 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-podinfo" pod "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" (UID: "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:38:13.926380 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.926355 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-token" pod "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" (UID: "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:13.926486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.926423 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-token" pod "b67abee5-46f6-464d-9a37-8afaf3ebdd5b" (UID: "b67abee5-46f6-464d-9a37-8afaf3ebdd5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:13.926583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.926563 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-podinfo" pod "b67abee5-46f6-464d-9a37-8afaf3ebdd5b" (UID: "b67abee5-46f6-464d-9a37-8afaf3ebdd5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:38:13.926687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.926670 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-kube-api-access-mmv8t" pod "b67abee5-46f6-464d-9a37-8afaf3ebdd5b" (UID: "b67abee5-46f6-464d-9a37-8afaf3ebdd5b"). InnerVolumeSpecName "kube-api-access-mmv8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:13.927570 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.927551 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-kube-api-access-vgxjx" pod "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" (UID: "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b"). InnerVolumeSpecName "kube-api-access-vgxjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:13.927635 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.927568 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-envoy" pod "b67abee5-46f6-464d-9a37-8afaf3ebdd5b" (UID: "b67abee5-46f6-464d-9a37-8afaf3ebdd5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:13.927969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:13.927929 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-envoy" pod "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" (UID: "b147f1b4-2219-4fcf-977b-e6b3b72b7c7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:14.026259 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.026227 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgxjx\" (UniqueName: \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-kube-api-access-vgxjx\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.026259 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.026260 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.026449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.026271 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.026449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.026279 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.026449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.026287 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.026449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.026296 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.026449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.026304 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.026449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.026312 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmv8t\" (UniqueName: \"kubernetes.io/projected/b67abee5-46f6-464d-9a37-8afaf3ebdd5b-kube-api-access-mmv8t\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.026449 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.026322 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.132019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.131986 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff"] Oct 30 22:38:14.132206 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.132183 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" containerID="cri-o://4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03" gracePeriod=30 Oct 30 22:38:14.363138 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.363100 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp"] Oct 30 22:38:14.367212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.367183 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-q5rsp"] Oct 30 22:38:14.479913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.479834 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq"] Oct 30 22:38:14.483735 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.483714 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-wrhzq"] Oct 30 22:38:14.741686 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.741592 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" path="/var/lib/kubelet/pods/b147f1b4-2219-4fcf-977b-e6b3b72b7c7b/volumes" Oct 30 22:38:14.742188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:14.742167 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" path="/var/lib/kubelet/pods/b67abee5-46f6-464d-9a37-8afaf3ebdd5b/volumes" Oct 30 22:38:18.931115 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:18.931078 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:38:18.931501 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:18.931140 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:38:19.415928 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.415903 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:38:19.494984 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.494952 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-token\" (UniqueName: \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-token\") pod \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " Oct 30 22:38:19.495142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.494996 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istiod-ca-cert\") pod \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " Oct 30 22:38:19.495142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495021 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-certs\") pod \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " Oct 30 22:38:19.495142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495075 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-credential-socket\") pod \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " Oct 30 22:38:19.495142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495111 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-data\") pod \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " Oct 30 22:38:19.495357 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495155 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-kube-api-access-6692p\" (UniqueName: \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-kube-api-access-6692p\") pod \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " Oct 30 22:38:19.495357 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495223 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-podinfo\") pod \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " Oct 30 22:38:19.495357 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495273 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-socket\") pod \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " Oct 30 22:38:19.495507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495359 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-envoy\") pod \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\" (UID: \"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd\") " Oct 30 22:38:19.495507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495441 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-credential-socket" pod "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" (UID: "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:19.495507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495480 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-data" pod "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" (UID: "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:19.495990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495717 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-certs" pod "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" (UID: "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:19.495990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495776 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-socket" pod "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" (UID: "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:19.495990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495912 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.495990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495930 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.495990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495972 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istiod-ca-cert" pod "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" (UID: "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:19.495990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.495974 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.496304 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.496003 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.497529 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.497501 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-token" pod "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" (UID: "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:19.497913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.497887 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-kube-api-access-6692p" pod "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" (UID: "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd"). InnerVolumeSpecName "kube-api-access-6692p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:19.498059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.497925 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-envoy" pod "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" (UID: "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:19.498059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.497970 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-podinfo" pod "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" (UID: "c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:38:19.597155 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.597120 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.597155 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.597151 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.597366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.597168 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6692p\" (UniqueName: \"kubernetes.io/projected/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-kube-api-access-6692p\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.597366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.597182 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.597366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.597195 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.886468 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.886430 4859 generic.go:356] "Generic (PLEG): container finished" podID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerID="4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03" exitCode=0 Oct 30 22:38:19.886655 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.886477 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" Oct 30 22:38:19.886655 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.886485 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" event={"ID":"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd","Type":"ContainerDied","Data":"4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03"} Oct 30 22:38:19.886655 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.886536 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff" event={"ID":"c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd","Type":"ContainerDied","Data":"253bdddc5213ef2f51767e6f20c01f501f311acf2f42d8e370d730d8b67cf274"} Oct 30 22:38:19.886655 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.886564 4859 scope.go:122] "RemoveContainer" containerID="4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03" Oct 30 22:38:19.897979 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.897963 4859 scope.go:122] "RemoveContainer" containerID="4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03" Oct 30 22:38:19.898265 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:38:19.898243 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03\": container with ID starting with 4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03 not found: ID does not exist" containerID="4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03" Oct 30 22:38:19.898363 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:19.898271 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03"} err="failed to get container status \"4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03\": rpc error: code = NotFound desc = could not find container \"4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03\": container with ID starting with 4f9cdedd2565d821a0c9638d670d406ccc50fa30f917ec78bab2e3273fa80d03 not found: ID does not exist" Oct 30 22:38:20.061004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:20.060935 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff"] Oct 30 22:38:20.110359 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:20.110323 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-pd2ff"] Oct 30 22:38:20.741646 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:20.741609 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" path="/var/lib/kubelet/pods/c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd/volumes" Oct 30 22:38:32.761478 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.761443 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6"] Oct 30 22:38:32.762009 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.761983 4859 cpu_manager.go:414] "containerMap: removing container" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerName="istio-proxy" Oct 30 22:38:32.762009 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.762000 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerName="istio-proxy" Oct 30 22:38:32.762209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.762020 4859 cpu_manager.go:414] "containerMap: removing container" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" Oct 30 22:38:32.762209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.762027 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" Oct 30 22:38:32.762209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.762073 4859 cpu_manager.go:414] "containerMap: removing container" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" Oct 30 22:38:32.762209 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.762081 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" Oct 30 22:38:32.762409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.762216 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b147f1b4-2219-4fcf-977b-e6b3b72b7c7b" containerName="istio-proxy" Oct 30 22:38:32.762409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.762237 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67abee5-46f6-464d-9a37-8afaf3ebdd5b" containerName="istio-proxy" Oct 30 22:38:32.762409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.762247 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76a03cc-7bd2-4fb9-a6a7-1df7f9577acd" containerName="istio-proxy" Oct 30 22:38:32.765589 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.765570 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.820789 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.820760 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9f7654\"/\"openshift-service-ca.crt\"" Oct 30 22:38:32.849693 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.849656 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-tmp-dir\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.849824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.849722 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-home\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.849987 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.849935 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf74k\" (UniqueName: \"kubernetes.io/projected/ea43b315-c85b-495b-b8a2-297d072c98e9-kube-api-access-rf74k\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.850179 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.850046 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-model-cache\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.850309 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.850201 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea43b315-c85b-495b-b8a2-297d072c98e9-tls-certs\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.850422 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.850402 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-dshm\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.869819 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.869789 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9f7654\"/\"kube-root-ca.crt\"" Oct 30 22:38:32.869933 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.869916 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9f7654\"/\"auth-test-model-9f7654-kserve-self-signed-certs\"" Oct 30 22:38:32.869933 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.869915 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9f7654\"/\"default-dockercfg-8v29v\"" Oct 30 22:38:32.873422 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.873402 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6"] Oct 30 22:38:32.951896 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.951863 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-dshm\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.952105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.951920 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-tmp-dir\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.952105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.952057 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-home\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.952105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.952096 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf74k\" (UniqueName: \"kubernetes.io/projected/ea43b315-c85b-495b-b8a2-297d072c98e9-kube-api-access-rf74k\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.952231 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.952146 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-model-cache\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.952806 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.952280 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea43b315-c85b-495b-b8a2-297d072c98e9-tls-certs\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.952806 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.952283 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-tmp-dir\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.952806 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.952414 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-home\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.952806 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.952491 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-model-cache\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.954444 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.954421 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-dshm\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:32.955004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:32.954985 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea43b315-c85b-495b-b8a2-297d072c98e9-tls-certs\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:33.010391 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:33.010365 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf74k\" (UniqueName: \"kubernetes.io/projected/ea43b315-c85b-495b-b8a2-297d072c98e9-kube-api-access-rf74k\") pod \"auth-test-model-9f7654-kserve-8558c659d5-9h8r6\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:33.077386 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:33.077335 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:33.455212 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:33.455183 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6"] Oct 30 22:38:33.455865 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:38:33.455842 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea43b315_c85b_495b_b8a2_297d072c98e9.slice/crio-a44ed8fcd0c20ff1298f5feff393d38e402c27e32f57611e29d6e9af68dfa033 WatchSource:0}: Error finding container a44ed8fcd0c20ff1298f5feff393d38e402c27e32f57611e29d6e9af68dfa033: Status 404 returned error can't find the container with id a44ed8fcd0c20ff1298f5feff393d38e402c27e32f57611e29d6e9af68dfa033 Oct 30 22:38:34.028193 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:34.028155 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" event={"ID":"ea43b315-c85b-495b-b8a2-297d072c98e9","Type":"ContainerStarted","Data":"e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1"} Oct 30 22:38:34.028557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:34.028206 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" event={"ID":"ea43b315-c85b-495b-b8a2-297d072c98e9","Type":"ContainerStarted","Data":"a44ed8fcd0c20ff1298f5feff393d38e402c27e32f57611e29d6e9af68dfa033"} Oct 30 22:38:34.078390 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:34.078365 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:34.143105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:34.143053 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" podStartSLOduration=2.143039426 podStartE2EDuration="2.143039426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:34.140500757 +0000 UTC m=+3371.513127503" watchObservedRunningTime="2026-10-30 22:38:34.143039426 +0000 UTC m=+3371.515666172" Oct 30 22:38:39.881999 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:38:39.881911 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:38:43.078451 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:43.078408 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:43.092407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:43.092379 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:43.103986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:43.103963 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:38:48.075266 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:38:48.075231 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:39:09.878558 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:39:09.878514 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:39:21.319423 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.319385 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq"] Oct 30 22:39:21.321936 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.319604 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" podUID="4d1f86d0-dacc-4d24-8425-d8a581152da8" containerName="maas-api" containerID="cri-o://fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83" gracePeriod=30 Oct 30 22:39:21.631539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.631506 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:39:21.659745 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.659709 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-metrics-certs\") pod \"4d1f86d0-dacc-4d24-8425-d8a581152da8\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " Oct 30 22:39:21.659939 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.659890 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-maas-api-tls\") pod \"4d1f86d0-dacc-4d24-8425-d8a581152da8\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " Oct 30 22:39:21.660030 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.659932 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d1f86d0-dacc-4d24-8425-d8a581152da8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1f86d0-dacc-4d24-8425-d8a581152da8-openshift-service-ca\") pod \"4d1f86d0-dacc-4d24-8425-d8a581152da8\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " Oct 30 22:39:21.660094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.660037 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1f86d0-dacc-4d24-8425-d8a581152da8-kube-api-access-742tj\" (UniqueName: \"kubernetes.io/projected/4d1f86d0-dacc-4d24-8425-d8a581152da8-kube-api-access-742tj\") pod \"4d1f86d0-dacc-4d24-8425-d8a581152da8\" (UID: \"4d1f86d0-dacc-4d24-8425-d8a581152da8\") " Oct 30 22:39:21.660333 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.660303 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1f86d0-dacc-4d24-8425-d8a581152da8-openshift-service-ca" pod "4d1f86d0-dacc-4d24-8425-d8a581152da8" (UID: "4d1f86d0-dacc-4d24-8425-d8a581152da8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:21.660526 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.660510 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1f86d0-dacc-4d24-8425-d8a581152da8-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:21.662121 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.662090 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1f86d0-dacc-4d24-8425-d8a581152da8-kube-api-access-742tj" pod "4d1f86d0-dacc-4d24-8425-d8a581152da8" (UID: "4d1f86d0-dacc-4d24-8425-d8a581152da8"). InnerVolumeSpecName "kube-api-access-742tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:21.662294 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.662270 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-metrics-certs" pod "4d1f86d0-dacc-4d24-8425-d8a581152da8" (UID: "4d1f86d0-dacc-4d24-8425-d8a581152da8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:21.662417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.662312 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-maas-api-tls" pod "4d1f86d0-dacc-4d24-8425-d8a581152da8" (UID: "4d1f86d0-dacc-4d24-8425-d8a581152da8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:21.778993 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.764886 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-742tj\" (UniqueName: \"kubernetes.io/projected/4d1f86d0-dacc-4d24-8425-d8a581152da8-kube-api-access-742tj\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:21.778993 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.764924 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:21.778993 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:21.764957 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1f86d0-dacc-4d24-8425-d8a581152da8-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:22.368795 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:22.368763 4859 generic.go:356] "Generic (PLEG): container finished" podID="4d1f86d0-dacc-4d24-8425-d8a581152da8" containerID="fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83" exitCode=0 Oct 30 22:39:22.369234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:22.368813 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" Oct 30 22:39:22.369234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:22.368815 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" event={"ID":"4d1f86d0-dacc-4d24-8425-d8a581152da8","Type":"ContainerDied","Data":"fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83"} Oct 30 22:39:22.369234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:22.368983 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq" event={"ID":"4d1f86d0-dacc-4d24-8425-d8a581152da8","Type":"ContainerDied","Data":"7e4796d2aea35fa6774ea240bd18c252e64ac673d8d7a9ca149e3053e1a12fa9"} Oct 30 22:39:22.369234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:22.369051 4859 scope.go:122] "RemoveContainer" containerID="fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83" Oct 30 22:39:22.381923 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:22.381899 4859 scope.go:122] "RemoveContainer" containerID="fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83" Oct 30 22:39:22.382261 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:39:22.382237 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83\": container with ID starting with fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83 not found: ID does not exist" containerID="fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83" Oct 30 22:39:22.382310 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:22.382271 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83"} err="failed to get container status \"fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83\": rpc error: code = NotFound desc = could not find container \"fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83\": container with ID starting with fc342af543a7c67e20f411b10b6268017b733363fbcca894d0f1db0744df7c83 not found: ID does not exist" Oct 30 22:39:22.408437 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:39:22.408408 4859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1f86d0_dacc_4d24_8425_d8a581152da8.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:39:22.535651 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:22.535617 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq"] Oct 30 22:39:22.539236 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:22.539210 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9651e9-59dc58785c-x49vq"] Oct 30 22:39:22.741714 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:22.741637 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d1f86d0-dacc-4d24-8425-d8a581152da8" path="/var/lib/kubelet/pods/4d1f86d0-dacc-4d24-8425-d8a581152da8/volumes" Oct 30 22:39:36.843966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:36.842976 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p"] Oct 30 22:39:36.843966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:36.843890 4859 cpu_manager.go:414] "containerMap: removing container" podUID="4d1f86d0-dacc-4d24-8425-d8a581152da8" containerName="maas-api" Oct 30 22:39:36.843966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:36.843913 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1f86d0-dacc-4d24-8425-d8a581152da8" containerName="maas-api" Oct 30 22:39:36.844606 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:36.844170 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1f86d0-dacc-4d24-8425-d8a581152da8" containerName="maas-api" Oct 30 22:39:36.855065 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:36.855039 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:36.913244 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:36.913062 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-6eae99-openshift-default-dockercfg-s5j28\"" Oct 30 22:39:37.012049 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.012014 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-envoy\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.012049 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.012052 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fad467-5997-4c97-8651-6e87287fd98e-istio-podinfo\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.012298 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.012114 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-data\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.012298 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.012136 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-istio-token\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.012298 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.012151 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-certs\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.012298 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.012179 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69pqs\" (UniqueName: \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-kube-api-access-69pqs\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.012298 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.012201 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-credential-socket\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.012298 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.012277 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fad467-5997-4c97-8651-6e87287fd98e-istiod-ca-cert\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.012610 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.012321 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-socket\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.019059 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.019028 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p"] Oct 30 22:39:37.113188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113099 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-data\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113157 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-istio-token\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113181 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-certs\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113419 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113224 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69pqs\" (UniqueName: \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-kube-api-access-69pqs\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113419 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113261 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-credential-socket\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113419 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113377 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fad467-5997-4c97-8651-6e87287fd98e-istiod-ca-cert\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113449 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-socket\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113503 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-envoy\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113526 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-data\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113529 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fad467-5997-4c97-8651-6e87287fd98e-istio-podinfo\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113862 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113840 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-certs\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.113968 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.113923 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-socket\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.114103 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.114085 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-credential-socket\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.114186 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.114168 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fad467-5997-4c97-8651-6e87287fd98e-istiod-ca-cert\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.115754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.115734 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-envoy\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.116053 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.116035 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fad467-5997-4c97-8651-6e87287fd98e-istio-podinfo\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.169928 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.169898 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-istio-token\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.170077 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.169978 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69pqs\" (UniqueName: \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-kube-api-access-69pqs\") pod \"e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.235052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.234700 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:37.677273 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:39:37.677241 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fad467_5997_4c97_8651_6e87287fd98e.slice/crio-959c12615c7eff16c9d232cec94623c960d3383444c9a54121c28d5e614ef9e3 WatchSource:0}: Error finding container 959c12615c7eff16c9d232cec94623c960d3383444c9a54121c28d5e614ef9e3: Status 404 returned error can't find the container with id 959c12615c7eff16c9d232cec94623c960d3383444c9a54121c28d5e614ef9e3 Oct 30 22:39:37.677392 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.677325 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p"] Oct 30 22:39:37.680114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.680071 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:39:37.680208 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.680163 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:39:37.680247 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:37.680207 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:39:38.492300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:38.492260 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" event={"ID":"23fad467-5997-4c97-8651-6e87287fd98e","Type":"ContainerStarted","Data":"9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e"} Oct 30 22:39:38.492708 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:38.492321 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" event={"ID":"23fad467-5997-4c97-8651-6e87287fd98e","Type":"ContainerStarted","Data":"959c12615c7eff16c9d232cec94623c960d3383444c9a54121c28d5e614ef9e3"} Oct 30 22:39:38.613339 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:38.613283 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" podStartSLOduration=2.613268203 podStartE2EDuration="2.613268203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:38.610294251 +0000 UTC m=+3435.982920997" watchObservedRunningTime="2026-10-30 22:39:38.613268203 +0000 UTC m=+3435.985895021" Oct 30 22:39:39.238145 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:39.238109 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:39.243842 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:39.243816 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:39.500566 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:39.500475 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:39.501628 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:39.501606 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:39:39.881624 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:39:39.881582 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:39:41.878752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:41.878716 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:39:59.949882 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:59.949842 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn"] Oct 30 22:39:59.950458 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:39:59.950063 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" containerID="cri-o://8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844" gracePeriod=30 Oct 30 22:40:05.220591 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.220534 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:40:05.295188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295151 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-envoy\") pod \"f95e7b7c-f119-4ead-9f6a-a170e854e421\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " Oct 30 22:40:05.295341 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295204 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-socket\") pod \"f95e7b7c-f119-4ead-9f6a-a170e854e421\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " Oct 30 22:40:05.295341 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295245 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-certs\") pod \"f95e7b7c-f119-4ead-9f6a-a170e854e421\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " Oct 30 22:40:05.295341 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295277 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95e7b7c-f119-4ead-9f6a-a170e854e421-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95e7b7c-f119-4ead-9f6a-a170e854e421-istiod-ca-cert\") pod \"f95e7b7c-f119-4ead-9f6a-a170e854e421\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " Oct 30 22:40:05.295515 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295368 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-token\" (UniqueName: \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-token\") pod \"f95e7b7c-f119-4ead-9f6a-a170e854e421\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " Oct 30 22:40:05.295515 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295401 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-data\") pod \"f95e7b7c-f119-4ead-9f6a-a170e854e421\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " Oct 30 22:40:05.295515 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295446 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-podinfo\") pod \"f95e7b7c-f119-4ead-9f6a-a170e854e421\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " Oct 30 22:40:05.295681 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295544 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-kube-api-access-4pwjt\" (UniqueName: \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-kube-api-access-4pwjt\") pod \"f95e7b7c-f119-4ead-9f6a-a170e854e421\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " Oct 30 22:40:05.295681 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295606 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-credential-socket\") pod \"f95e7b7c-f119-4ead-9f6a-a170e854e421\" (UID: \"f95e7b7c-f119-4ead-9f6a-a170e854e421\") " Oct 30 22:40:05.295844 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295715 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-data" pod "f95e7b7c-f119-4ead-9f6a-a170e854e421" (UID: "f95e7b7c-f119-4ead-9f6a-a170e854e421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:05.295844 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295769 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95e7b7c-f119-4ead-9f6a-a170e854e421-istiod-ca-cert" pod "f95e7b7c-f119-4ead-9f6a-a170e854e421" (UID: "f95e7b7c-f119-4ead-9f6a-a170e854e421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:05.295844 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.295772 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-certs" pod "f95e7b7c-f119-4ead-9f6a-a170e854e421" (UID: "f95e7b7c-f119-4ead-9f6a-a170e854e421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:05.296091 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.296033 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-socket" pod "f95e7b7c-f119-4ead-9f6a-a170e854e421" (UID: "f95e7b7c-f119-4ead-9f6a-a170e854e421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:05.296169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.296103 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-credential-socket" pod "f95e7b7c-f119-4ead-9f6a-a170e854e421" (UID: "f95e7b7c-f119-4ead-9f6a-a170e854e421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:05.296169 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.296146 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.296272 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.296167 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.296272 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.296182 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.296272 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.296194 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95e7b7c-f119-4ead-9f6a-a170e854e421-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.297779 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.297753 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-envoy" pod "f95e7b7c-f119-4ead-9f6a-a170e854e421" (UID: "f95e7b7c-f119-4ead-9f6a-a170e854e421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:05.298170 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.298144 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-podinfo" pod "f95e7b7c-f119-4ead-9f6a-a170e854e421" (UID: "f95e7b7c-f119-4ead-9f6a-a170e854e421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:05.298286 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.298170 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-token" pod "f95e7b7c-f119-4ead-9f6a-a170e854e421" (UID: "f95e7b7c-f119-4ead-9f6a-a170e854e421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:05.298286 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.298261 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-kube-api-access-4pwjt" pod "f95e7b7c-f119-4ead-9f6a-a170e854e421" (UID: "f95e7b7c-f119-4ead-9f6a-a170e854e421"). InnerVolumeSpecName "kube-api-access-4pwjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:05.397258 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.397229 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.397258 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.397254 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.397258 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.397262 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.397513 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.397271 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95e7b7c-f119-4ead-9f6a-a170e854e421-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.397513 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.397280 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pwjt\" (UniqueName: \"kubernetes.io/projected/f95e7b7c-f119-4ead-9f6a-a170e854e421-kube-api-access-4pwjt\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.685054 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.685019 4859 generic.go:356] "Generic (PLEG): container finished" podID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerID="8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844" exitCode=0 Oct 30 22:40:05.685241 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.685074 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" Oct 30 22:40:05.685241 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.685107 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" event={"ID":"f95e7b7c-f119-4ead-9f6a-a170e854e421","Type":"ContainerDied","Data":"8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844"} Oct 30 22:40:05.685241 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.685163 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" event={"ID":"f95e7b7c-f119-4ead-9f6a-a170e854e421","Type":"ContainerDied","Data":"90d07a19e702f344be1c96cd2d0f2ad1d383447a1a228228b38c7c074f867bfa"} Oct 30 22:40:05.685241 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.685186 4859 scope.go:122] "RemoveContainer" containerID="8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844" Oct 30 22:40:05.695997 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.695986 4859 scope.go:122] "RemoveContainer" containerID="8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844" Oct 30 22:40:05.696262 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:40:05.696244 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844\": container with ID starting with 8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844 not found: ID does not exist" containerID="8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844" Oct 30 22:40:05.696320 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.696270 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844"} err="failed to get container status \"8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844\": rpc error: code = NotFound desc = could not find container \"8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844\": container with ID starting with 8481c573410bb482a532d7f409f0a7457c68a883cac2723ae7f7b5dc6b119844 not found: ID does not exist" Oct 30 22:40:05.850019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.849985 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn"] Oct 30 22:40:05.904498 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:05.904461 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn"] Oct 30 22:40:06.216302 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:06.216266 4859 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:40:06.216463 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:06.216321 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-z6dwn" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:40:06.740936 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:06.740900 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" path="/var/lib/kubelet/pods/f95e7b7c-f119-4ead-9f6a-a170e854e421/volumes" Oct 30 22:40:09.880062 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:40:09.880025 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:40:20.702435 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:40:20.702399 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:40:39.886413 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:40:39.886362 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:41:09.876881 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:41:09.876844 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:41:15.897809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:15.897772 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p"] Oct 30 22:41:15.898474 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:15.898009 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" podUID="23fad467-5997-4c97-8651-6e87287fd98e" containerName="istio-proxy" containerID="cri-o://9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e" gracePeriod=30 Oct 30 22:41:18.728625 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:18.728568 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:41:21.170826 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.170764 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:41:21.245117 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.245082 4859 generic.go:356] "Generic (PLEG): container finished" podID="23fad467-5997-4c97-8651-6e87287fd98e" containerID="9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e" exitCode=0 Oct 30 22:41:21.245273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.245140 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" Oct 30 22:41:21.245273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.245170 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" event={"ID":"23fad467-5997-4c97-8651-6e87287fd98e","Type":"ContainerDied","Data":"9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e"} Oct 30 22:41:21.245273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.245235 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p" event={"ID":"23fad467-5997-4c97-8651-6e87287fd98e","Type":"ContainerDied","Data":"959c12615c7eff16c9d232cec94623c960d3383444c9a54121c28d5e614ef9e3"} Oct 30 22:41:21.245273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.245260 4859 scope.go:122] "RemoveContainer" containerID="9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e" Oct 30 22:41:21.254983 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.254966 4859 scope.go:122] "RemoveContainer" containerID="9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e" Oct 30 22:41:21.255253 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:41:21.255235 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e\": container with ID starting with 9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e not found: ID does not exist" containerID="9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e" Oct 30 22:41:21.255308 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.255263 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e"} err="failed to get container status \"9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e\": rpc error: code = NotFound desc = could not find container \"9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e\": container with ID starting with 9f43a06b79d498344e4f64f7d810f2bf7d3d006806526216b82b54d24afddb7e not found: ID does not exist" Oct 30 22:41:21.332624 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.332588 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-data\") pod \"23fad467-5997-4c97-8651-6e87287fd98e\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " Oct 30 22:41:21.332624 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.332627 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-envoy\") pod \"23fad467-5997-4c97-8651-6e87287fd98e\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " Oct 30 22:41:21.332833 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.332656 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-certs\") pod \"23fad467-5997-4c97-8651-6e87287fd98e\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " Oct 30 22:41:21.332833 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.332725 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fad467-5997-4c97-8651-6e87287fd98e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fad467-5997-4c97-8651-6e87287fd98e-istiod-ca-cert\") pod \"23fad467-5997-4c97-8651-6e87287fd98e\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " Oct 30 22:41:21.332833 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.332774 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-kube-api-access-69pqs\" (UniqueName: \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-kube-api-access-69pqs\") pod \"23fad467-5997-4c97-8651-6e87287fd98e\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " Oct 30 22:41:21.332833 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.332827 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-socket\") pod \"23fad467-5997-4c97-8651-6e87287fd98e\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " Oct 30 22:41:21.333036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.332935 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-data" pod "23fad467-5997-4c97-8651-6e87287fd98e" (UID: "23fad467-5997-4c97-8651-6e87287fd98e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:21.333036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.332977 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23fad467-5997-4c97-8651-6e87287fd98e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fad467-5997-4c97-8651-6e87287fd98e-istio-podinfo\") pod \"23fad467-5997-4c97-8651-6e87287fd98e\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " Oct 30 22:41:21.333140 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333028 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-certs" pod "23fad467-5997-4c97-8651-6e87287fd98e" (UID: "23fad467-5997-4c97-8651-6e87287fd98e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:21.333140 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333038 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-istio-token\" (UniqueName: \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-istio-token\") pod \"23fad467-5997-4c97-8651-6e87287fd98e\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " Oct 30 22:41:21.333140 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333134 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-credential-socket\") pod \"23fad467-5997-4c97-8651-6e87287fd98e\" (UID: \"23fad467-5997-4c97-8651-6e87287fd98e\") " Oct 30 22:41:21.333288 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333256 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fad467-5997-4c97-8651-6e87287fd98e-istiod-ca-cert" pod "23fad467-5997-4c97-8651-6e87287fd98e" (UID: "23fad467-5997-4c97-8651-6e87287fd98e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:21.333383 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333361 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-socket" pod "23fad467-5997-4c97-8651-6e87287fd98e" (UID: "23fad467-5997-4c97-8651-6e87287fd98e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:21.333586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333541 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-credential-socket" pod "23fad467-5997-4c97-8651-6e87287fd98e" (UID: "23fad467-5997-4c97-8651-6e87287fd98e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:21.333726 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333715 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:21.333795 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333735 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:21.333795 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333750 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:21.333795 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333766 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fad467-5997-4c97-8651-6e87287fd98e-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:21.333795 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.333779 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:21.335035 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.335016 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-envoy" pod "23fad467-5997-4c97-8651-6e87287fd98e" (UID: "23fad467-5997-4c97-8651-6e87287fd98e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:21.335164 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.335060 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23fad467-5997-4c97-8651-6e87287fd98e-istio-podinfo" pod "23fad467-5997-4c97-8651-6e87287fd98e" (UID: "23fad467-5997-4c97-8651-6e87287fd98e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:21.335512 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.335476 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-kube-api-access-69pqs" pod "23fad467-5997-4c97-8651-6e87287fd98e" (UID: "23fad467-5997-4c97-8651-6e87287fd98e"). InnerVolumeSpecName "kube-api-access-69pqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:21.335686 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.335630 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-istio-token" pod "23fad467-5997-4c97-8651-6e87287fd98e" (UID: "23fad467-5997-4c97-8651-6e87287fd98e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:21.434664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.434576 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fad467-5997-4c97-8651-6e87287fd98e-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:21.434664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.434607 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:21.434664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.434622 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fad467-5997-4c97-8651-6e87287fd98e-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:21.434664 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.434634 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69pqs\" (UniqueName: \"kubernetes.io/projected/23fad467-5997-4c97-8651-6e87287fd98e-kube-api-access-69pqs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:21.670041 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.670002 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p"] Oct 30 22:41:21.673230 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:21.673204 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-6eae99-openshift-default-847d7f8fcf-t7f8p"] Oct 30 22:41:22.742326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:22.742296 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fad467-5997-4c97-8651-6e87287fd98e" path="/var/lib/kubelet/pods/23fad467-5997-4c97-8651-6e87287fd98e/volumes" Oct 30 22:41:25.746645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:25.746613 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx"] Oct 30 22:41:25.747067 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:25.746813 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerName="istio-proxy" containerID="cri-o://0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed" gracePeriod=30 Oct 30 22:41:31.015607 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.015532 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:41:31.065107 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065072 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-envoy\") pod \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " Oct 30 22:41:31.065296 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065166 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-credential-socket\") pod \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " Oct 30 22:41:31.065296 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065193 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-certs\") pod \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " Oct 30 22:41:31.065432 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065311 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-podinfo\") pod \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " Oct 30 22:41:31.065432 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065359 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-data\") pod \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " Oct 30 22:41:31.065432 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065386 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istiod-ca-cert\") pod \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " Oct 30 22:41:31.065588 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065507 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-certs" pod "f5f748cb-e716-42f1-ba4e-cfd4aca65d04" (UID: "f5f748cb-e716-42f1-ba4e-cfd4aca65d04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:31.065693 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065665 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-socket\") pod \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " Oct 30 22:41:31.065818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065741 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-kube-api-access-99dtb\" (UniqueName: \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-kube-api-access-99dtb\") pod \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " Oct 30 22:41:31.065818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065777 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-token\" (UniqueName: \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-token\") pod \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\" (UID: \"f5f748cb-e716-42f1-ba4e-cfd4aca65d04\") " Oct 30 22:41:31.065818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065794 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istiod-ca-cert" pod "f5f748cb-e716-42f1-ba4e-cfd4aca65d04" (UID: "f5f748cb-e716-42f1-ba4e-cfd4aca65d04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:31.066000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.065978 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-socket" pod "f5f748cb-e716-42f1-ba4e-cfd4aca65d04" (UID: "f5f748cb-e716-42f1-ba4e-cfd4aca65d04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:31.066325 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.066212 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-data" pod "f5f748cb-e716-42f1-ba4e-cfd4aca65d04" (UID: "f5f748cb-e716-42f1-ba4e-cfd4aca65d04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:31.066325 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.066297 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-credential-socket" pod "f5f748cb-e716-42f1-ba4e-cfd4aca65d04" (UID: "f5f748cb-e716-42f1-ba4e-cfd4aca65d04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:31.066325 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.066304 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.066325 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.066327 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.066612 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.066342 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.068231 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.068209 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-envoy" pod "f5f748cb-e716-42f1-ba4e-cfd4aca65d04" (UID: "f5f748cb-e716-42f1-ba4e-cfd4aca65d04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:31.068379 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.068315 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-kube-api-access-99dtb" pod "f5f748cb-e716-42f1-ba4e-cfd4aca65d04" (UID: "f5f748cb-e716-42f1-ba4e-cfd4aca65d04"). InnerVolumeSpecName "kube-api-access-99dtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:31.068712 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.068688 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-token" pod "f5f748cb-e716-42f1-ba4e-cfd4aca65d04" (UID: "f5f748cb-e716-42f1-ba4e-cfd4aca65d04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:31.068810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.068740 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-podinfo" pod "f5f748cb-e716-42f1-ba4e-cfd4aca65d04" (UID: "f5f748cb-e716-42f1-ba4e-cfd4aca65d04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:31.167577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.167537 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.167577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.167571 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.167577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.167582 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99dtb\" (UniqueName: \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-kube-api-access-99dtb\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.167788 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.167593 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.167788 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.167601 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.167788 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.167611 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f748cb-e716-42f1-ba4e-cfd4aca65d04-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.328132 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.328041 4859 generic.go:356] "Generic (PLEG): container finished" podID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerID="0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed" exitCode=0 Oct 30 22:41:31.328132 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.328087 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" Oct 30 22:41:31.328132 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.328098 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" event={"ID":"f5f748cb-e716-42f1-ba4e-cfd4aca65d04","Type":"ContainerDied","Data":"0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed"} Oct 30 22:41:31.328433 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.328153 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx" event={"ID":"f5f748cb-e716-42f1-ba4e-cfd4aca65d04","Type":"ContainerDied","Data":"b94cca08eb1c98ff16d6316dcd61115933f0c2afca2b23dc3672d07acb313923"} Oct 30 22:41:31.328433 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.328181 4859 scope.go:122] "RemoveContainer" containerID="0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed" Oct 30 22:41:31.339183 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.339155 4859 scope.go:122] "RemoveContainer" containerID="0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed" Oct 30 22:41:31.339493 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:41:31.339472 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed\": container with ID starting with 0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed not found: ID does not exist" containerID="0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed" Oct 30 22:41:31.339541 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.339505 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed"} err="failed to get container status \"0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed\": rpc error: code = NotFound desc = could not find container \"0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed\": container with ID starting with 0bea39a6e76b53adae0fdc07e05e5b9f456911257823099ebb6be1c6597022ed not found: ID does not exist" Oct 30 22:41:31.493404 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.493362 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx"] Oct 30 22:41:31.497038 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:31.497004 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-cbhnx"] Oct 30 22:41:32.741498 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:41:32.741458 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" path="/var/lib/kubelet/pods/f5f748cb-e716-42f1-ba4e-cfd4aca65d04/volumes" Oct 30 22:41:39.879239 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:41:39.879140 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:42:03.922238 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.922190 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8"] Oct 30 22:42:03.925875 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.925840 4859 cpu_manager.go:414] "containerMap: removing container" podUID="23fad467-5997-4c97-8651-6e87287fd98e" containerName="istio-proxy" Oct 30 22:42:03.926166 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.926144 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fad467-5997-4c97-8651-6e87287fd98e" containerName="istio-proxy" Oct 30 22:42:03.926345 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.926330 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" Oct 30 22:42:03.929964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.926424 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" Oct 30 22:42:03.930242 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.930218 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerName="istio-proxy" Oct 30 22:42:03.930341 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.930328 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerName="istio-proxy" Oct 30 22:42:03.930691 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.930670 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95e7b7c-f119-4ead-9f6a-a170e854e421" containerName="istio-proxy" Oct 30 22:42:03.945119 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.945082 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f748cb-e716-42f1-ba4e-cfd4aca65d04" containerName="istio-proxy" Oct 30 22:42:03.945412 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.945394 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fad467-5997-4c97-8651-6e87287fd98e" containerName="istio-proxy" Oct 30 22:42:03.949220 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.949192 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:03.989972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:03.986629 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8"] Oct 30 22:42:04.027981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.027927 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a6900c-openshift-default-dockercfg-b5zn2\"" Oct 30 22:42:04.072441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.072396 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-token\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.072650 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.072567 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-podinfo\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.072731 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.072666 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-socket\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.072775 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.072724 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-certs\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.072828 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.072781 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5df7\" (UniqueName: \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-kube-api-access-w5df7\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.072909 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.072874 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-credential-socket\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.072977 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.072925 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istiod-ca-cert\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.073080 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.073064 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-envoy\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.073126 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.073093 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-data\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.174377 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.174280 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-podinfo\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.174548 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.174447 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-socket\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.174548 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.174476 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-certs\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.174548 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.174525 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5df7\" (UniqueName: \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-kube-api-access-w5df7\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.174715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.174553 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-credential-socket\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.174715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.174587 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istiod-ca-cert\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.174715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.174641 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-envoy\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.174715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.174668 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-data\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.174936 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.174755 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-token\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.175151 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.175123 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-certs\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.175302 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.175228 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-data\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.175375 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.175294 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-socket\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.175561 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.175538 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istiod-ca-cert\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.175634 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.175557 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-credential-socket\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.177483 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.177458 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-podinfo\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.177679 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.177657 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-envoy\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.233627 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.233591 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-token\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.236210 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.236166 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5df7\" (UniqueName: \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-kube-api-access-w5df7\") pod \"e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.280765 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.280728 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:04.715118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.715080 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8"] Oct 30 22:42:04.715767 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:42:04.715741 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825d33d5_6db2_4c6b_a8a9_bd5a8a615449.slice/crio-cd876bd590f0364fe32164e9e6331bdce74af2218283f7dc1ce9f6c09252f1a3 WatchSource:0}: Error finding container cd876bd590f0364fe32164e9e6331bdce74af2218283f7dc1ce9f6c09252f1a3: Status 404 returned error can't find the container with id cd876bd590f0364fe32164e9e6331bdce74af2218283f7dc1ce9f6c09252f1a3 Oct 30 22:42:04.718330 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.718299 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:42:04.718407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.718365 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:42:04.718407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:04.718393 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:42:05.585371 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:05.585330 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" event={"ID":"825d33d5-6db2-4c6b-a8a9-bd5a8a615449","Type":"ContainerStarted","Data":"5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a"} Oct 30 22:42:05.585894 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:05.585396 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" event={"ID":"825d33d5-6db2-4c6b-a8a9-bd5a8a615449","Type":"ContainerStarted","Data":"cd876bd590f0364fe32164e9e6331bdce74af2218283f7dc1ce9f6c09252f1a3"} Oct 30 22:42:05.754809 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:05.754734 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" podStartSLOduration=2.754712493 podStartE2EDuration="2.754712493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:05.752838949 +0000 UTC m=+3583.125465719" watchObservedRunningTime="2026-10-30 22:42:05.754712493 +0000 UTC m=+3583.127339244" Oct 30 22:42:06.281862 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:06.281816 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:06.288021 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:06.287991 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:06.605691 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:06.605654 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:06.606873 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:06.606845 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:42:09.881510 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:42:09.881465 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:42:12.061158 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:12.061116 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:42:23.399420 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.399394 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:42:23.407536 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.407509 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:42:23.422888 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.422867 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:42:23.429872 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.429854 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:42:23.840186 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.840152 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf"] Oct 30 22:42:23.853729 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.853702 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:23.940904 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.940839 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf"] Oct 30 22:42:23.950171 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.950136 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-20ccef-openshift-default-dockercfg-fgmll\"" Oct 30 22:42:23.951891 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.951848 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-socket\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:23.952244 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.952213 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-token\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:23.952549 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.952527 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-data\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:23.952836 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.952795 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-envoy\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:23.953098 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.953070 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfk7z\" (UniqueName: \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-kube-api-access-cfk7z\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:23.953333 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.953313 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-podinfo\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:23.953558 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.953540 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istiod-ca-cert\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:23.953765 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.953745 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-credential-socket\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:23.954016 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:23.953995 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-certs\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056046 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-data\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056207 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056148 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-envoy\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056207 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056183 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfk7z\" (UniqueName: \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-kube-api-access-cfk7z\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056328 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056208 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-podinfo\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056328 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056258 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istiod-ca-cert\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056328 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056284 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-credential-socket\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056328 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056315 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-certs\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056536 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056439 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-socket\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056536 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056468 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-token\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056536 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056496 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-data\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056856 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056819 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-certs\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.056856 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.056842 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-socket\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.057142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.057046 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-credential-socket\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.057142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.057124 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istiod-ca-cert\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.058634 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.058615 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-envoy\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.059451 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.059435 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-podinfo\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.114155 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.114092 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-token\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.114155 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.114127 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfk7z\" (UniqueName: \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-kube-api-access-cfk7z\") pod \"e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.224723 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.224687 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:24.670255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.670228 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf"] Oct 30 22:42:24.670847 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:42:24.670827 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f42837_1b1c_4f5b_9236_16ed9c6cb95b.slice/crio-75e6c3adf80e337b9880b58ab7bea44c855c67e8f736a271ae7346e10b95dea5 WatchSource:0}: Error finding container 75e6c3adf80e337b9880b58ab7bea44c855c67e8f736a271ae7346e10b95dea5: Status 404 returned error can't find the container with id 75e6c3adf80e337b9880b58ab7bea44c855c67e8f736a271ae7346e10b95dea5 Oct 30 22:42:24.673715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.673682 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:42:24.673821 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.673748 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:42:24.673821 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.673778 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:42:24.744385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:24.744354 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" event={"ID":"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b","Type":"ContainerStarted","Data":"75e6c3adf80e337b9880b58ab7bea44c855c67e8f736a271ae7346e10b95dea5"} Oct 30 22:42:25.754032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:25.753995 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" event={"ID":"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b","Type":"ContainerStarted","Data":"b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4"} Oct 30 22:42:25.950495 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:25.950443 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" podStartSLOduration=2.950430891 podStartE2EDuration="2.950430891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:25.949182301 +0000 UTC m=+3603.321809047" watchObservedRunningTime="2026-10-30 22:42:25.950430891 +0000 UTC m=+3603.323057640" Oct 30 22:42:26.225105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:26.225073 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:26.230713 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:26.230684 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:26.762200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:26.762166 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:26.763278 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:42:26.763252 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:42:39.877558 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:42:39.877515 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:43:09.887026 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:43:09.884203 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:43:10.272582 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:43:10.272492 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:43:39.882460 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:43:39.882424 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:44:09.880221 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:44:09.880184 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:44:10.758192 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:10.758159 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:44:22.647084 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:22.647046 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:44:22.659776 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:22.647565 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:44:23.153768 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.153740 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6"] Oct 30 22:44:23.728301 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.728265 4859 generic.go:356] "Generic (PLEG): container finished" podID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerID="5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a" exitCode=0 Oct 30 22:44:23.728754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.728348 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" event={"ID":"825d33d5-6db2-4c6b-a8a9-bd5a8a615449","Type":"ContainerDied","Data":"5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a"} Oct 30 22:44:23.728864 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.728843 4859 scope.go:122] "RemoveContainer" containerID="5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a" Oct 30 22:44:23.730142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.730118 4859 generic.go:356] "Generic (PLEG): container finished" podID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerID="b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4" exitCode=0 Oct 30 22:44:23.730221 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.730144 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" event={"ID":"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b","Type":"ContainerDied","Data":"b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4"} Oct 30 22:44:23.730490 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.730476 4859 scope.go:122] "RemoveContainer" containerID="b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4" Oct 30 22:44:23.730697 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.730669 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:44:23.730754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.730735 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:44:23.730807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.730766 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:44:23.731889 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.731865 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:44:23.731962 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.731923 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:44:23.732005 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:23.731981 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:44:24.742964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:24.742094 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" event={"ID":"825d33d5-6db2-4c6b-a8a9-bd5a8a615449","Type":"ContainerStarted","Data":"bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57"} Oct 30 22:44:24.744188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:24.744155 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" event={"ID":"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b","Type":"ContainerStarted","Data":"20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a"} Oct 30 22:44:25.225434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:25.225392 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:44:25.226936 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:25.226915 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:44:25.227003 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:25.226973 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:44:25.284380 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:25.284311 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:44:25.286084 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:25.286052 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 22:44:25.286337 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:25.286306 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 22:44:26.225552 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:26.225513 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:44:26.225985 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:26.225571 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:44:26.282076 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:26.282042 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 22:44:26.282867 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:26.282835 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 22:44:26.786974 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:26.779049 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:44:27.225627 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:27.225591 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:44:27.226034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:27.225647 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:44:27.289976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:27.288778 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 22:44:27.289976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:27.288833 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 22:44:28.225599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:28.225558 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:44:28.225791 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:28.225638 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:44:28.290330 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:28.290295 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 22:44:28.291137 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:28.291105 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 22:44:29.229417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:29.229382 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:44:29.229921 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:29.229896 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:44:29.315647 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:29.315613 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 22:44:29.315883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:29.315850 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 22:44:30.229587 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:30.229552 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:44:30.230476 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:30.230455 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:44:30.286239 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:30.286209 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:44:30.286608 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:30.286584 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:44:30.287319 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:30.287300 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:44:38.173865 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.173768 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" podUID="ea43b315-c85b-495b-b8a2-297d072c98e9" containerName="main" containerID="cri-o://e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1" gracePeriod=45 Oct 30 22:44:38.464584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.464555 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:44:38.595581 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.595538 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea43b315-c85b-495b-b8a2-297d072c98e9-kube-api-access-rf74k\" (UniqueName: \"kubernetes.io/projected/ea43b315-c85b-495b-b8a2-297d072c98e9-kube-api-access-rf74k\") pod \"ea43b315-c85b-495b-b8a2-297d072c98e9\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " Oct 30 22:44:38.595893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.595598 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-home\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-home\") pod \"ea43b315-c85b-495b-b8a2-297d072c98e9\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " Oct 30 22:44:38.595893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.595655 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-tmp-dir\") pod \"ea43b315-c85b-495b-b8a2-297d072c98e9\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " Oct 30 22:44:38.595893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.595722 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-dshm\") pod \"ea43b315-c85b-495b-b8a2-297d072c98e9\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " Oct 30 22:44:38.595893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.595824 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-model-cache\") pod \"ea43b315-c85b-495b-b8a2-297d072c98e9\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " Oct 30 22:44:38.595893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.595885 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea43b315-c85b-495b-b8a2-297d072c98e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea43b315-c85b-495b-b8a2-297d072c98e9-tls-certs\") pod \"ea43b315-c85b-495b-b8a2-297d072c98e9\" (UID: \"ea43b315-c85b-495b-b8a2-297d072c98e9\") " Oct 30 22:44:38.596752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.596446 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-tmp-dir" pod "ea43b315-c85b-495b-b8a2-297d072c98e9" (UID: "ea43b315-c85b-495b-b8a2-297d072c98e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:38.597501 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.597440 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-home" pod "ea43b315-c85b-495b-b8a2-297d072c98e9" (UID: "ea43b315-c85b-495b-b8a2-297d072c98e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:38.597602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.597509 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-model-cache" pod "ea43b315-c85b-495b-b8a2-297d072c98e9" (UID: "ea43b315-c85b-495b-b8a2-297d072c98e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:38.609140 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.608931 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-dshm" pod "ea43b315-c85b-495b-b8a2-297d072c98e9" (UID: "ea43b315-c85b-495b-b8a2-297d072c98e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:38.609140 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.609109 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea43b315-c85b-495b-b8a2-297d072c98e9-tls-certs" pod "ea43b315-c85b-495b-b8a2-297d072c98e9" (UID: "ea43b315-c85b-495b-b8a2-297d072c98e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:44:38.609262 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.609141 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea43b315-c85b-495b-b8a2-297d072c98e9-kube-api-access-rf74k" pod "ea43b315-c85b-495b-b8a2-297d072c98e9" (UID: "ea43b315-c85b-495b-b8a2-297d072c98e9"). InnerVolumeSpecName "kube-api-access-rf74k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:44:38.697572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.697491 4859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-model-cache\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:38.697572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.697525 4859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea43b315-c85b-495b-b8a2-297d072c98e9-tls-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:38.697572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.697538 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf74k\" (UniqueName: \"kubernetes.io/projected/ea43b315-c85b-495b-b8a2-297d072c98e9-kube-api-access-rf74k\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:38.697572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.697547 4859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-home\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:38.697572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.697576 4859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-tmp-dir\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:38.697927 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.697584 4859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea43b315-c85b-495b-b8a2-297d072c98e9-dshm\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:38.921414 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.921376 4859 generic.go:356] "Generic (PLEG): container finished" podID="ea43b315-c85b-495b-b8a2-297d072c98e9" containerID="e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1" exitCode=0 Oct 30 22:44:38.921597 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.921423 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" Oct 30 22:44:38.921597 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.921459 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" event={"ID":"ea43b315-c85b-495b-b8a2-297d072c98e9","Type":"ContainerDied","Data":"e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1"} Oct 30 22:44:38.921597 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.921521 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6" event={"ID":"ea43b315-c85b-495b-b8a2-297d072c98e9","Type":"ContainerDied","Data":"a44ed8fcd0c20ff1298f5feff393d38e402c27e32f57611e29d6e9af68dfa033"} Oct 30 22:44:38.921597 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.921543 4859 scope.go:122] "RemoveContainer" containerID="e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1" Oct 30 22:44:38.933116 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.933096 4859 scope.go:122] "RemoveContainer" containerID="e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1" Oct 30 22:44:38.933447 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:44:38.933426 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1\": container with ID starting with e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1 not found: ID does not exist" containerID="e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1" Oct 30 22:44:38.933572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:38.933456 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1"} err="failed to get container status \"e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1\": rpc error: code = NotFound desc = could not find container \"e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1\": container with ID starting with e0f8b4340e7401b848ac42459124a693dcd574c3c8fe4f5e26ff2b56fd45d6f1 not found: ID does not exist" Oct 30 22:44:39.092255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:39.092158 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6"] Oct 30 22:44:39.141396 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:39.141371 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9f7654/auth-test-model-9f7654-kserve-8558c659d5-9h8r6"] Oct 30 22:44:39.877546 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:44:39.877514 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:44:40.742093 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:40.742059 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea43b315-c85b-495b-b8a2-297d072c98e9" path="/var/lib/kubelet/pods/ea43b315-c85b-495b-b8a2-297d072c98e9/volumes" Oct 30 22:44:45.959823 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:45.959787 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9"] Oct 30 22:44:45.960406 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:45.960391 4859 cpu_manager.go:414] "containerMap: removing container" podUID="ea43b315-c85b-495b-b8a2-297d072c98e9" containerName="main" Oct 30 22:44:45.960452 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:45.960409 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea43b315-c85b-495b-b8a2-297d072c98e9" containerName="main" Oct 30 22:44:45.960525 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:45.960516 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea43b315-c85b-495b-b8a2-297d072c98e9" containerName="main" Oct 30 22:44:45.963377 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:45.963359 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.015465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.015439 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-15e0f6-serving-cert\"" Oct 30 22:44:46.070229 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.070194 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9"] Oct 30 22:44:46.076166 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.076041 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-maas-api-tls\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.076166 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.076081 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wkf\" (UniqueName: \"kubernetes.io/projected/336cf7be-cb8c-4938-b19e-e37f6f99b165-kube-api-access-77wkf\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.076166 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.076132 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336cf7be-cb8c-4938-b19e-e37f6f99b165-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.076435 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.076322 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-metrics-certs\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.177875 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.177844 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-metrics-certs\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.178075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.177906 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-maas-api-tls\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.178075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.177924 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wkf\" (UniqueName: \"kubernetes.io/projected/336cf7be-cb8c-4938-b19e-e37f6f99b165-kube-api-access-77wkf\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.178075 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:44:46.178039 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-serving-cert: secret "maas-api-e2e-extmod-a-15e0f6-serving-cert" not found Oct 30 22:44:46.178075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.178056 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336cf7be-cb8c-4938-b19e-e37f6f99b165-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.178308 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:44:46.178122 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-maas-api-tls podName:336cf7be-cb8c-4938-b19e-e37f6f99b165 nodeName:}" failed. No retries permitted until 2026-10-30 22:44:46.678101923 +0000 UTC m=+3744.050728659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-maas-api-tls") pod "maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" (UID: "336cf7be-cb8c-4938-b19e-e37f6f99b165") : secret "maas-api-e2e-extmod-a-15e0f6-serving-cert" not found Oct 30 22:44:46.178704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.178686 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336cf7be-cb8c-4938-b19e-e37f6f99b165-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.180309 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.180289 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-metrics-certs\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.245706 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.245619 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wkf\" (UniqueName: \"kubernetes.io/projected/336cf7be-cb8c-4938-b19e-e37f6f99b165-kube-api-access-77wkf\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.685096 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.685057 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-maas-api-tls\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.687489 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.687459 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-maas-api-tls\") pod \"maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:46.875243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:46.875214 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:47.267607 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:47.267574 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9"] Oct 30 22:44:47.268142 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:44:47.268119 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336cf7be_cb8c_4938_b19e_e37f6f99b165.slice/crio-a91ba391a03515d538231364a1a8fa147eb4d6ad2ea015c4647ad557df30113b WatchSource:0}: Error finding container a91ba391a03515d538231364a1a8fa147eb4d6ad2ea015c4647ad557df30113b: Status 404 returned error can't find the container with id a91ba391a03515d538231364a1a8fa147eb4d6ad2ea015c4647ad557df30113b Oct 30 22:44:47.269154 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:47.269133 4859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:44:47.984619 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:47.984587 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" event={"ID":"336cf7be-cb8c-4938-b19e-e37f6f99b165","Type":"ContainerStarted","Data":"a91ba391a03515d538231364a1a8fa147eb4d6ad2ea015c4647ad557df30113b"} Oct 30 22:44:48.992747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:48.992712 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" event={"ID":"336cf7be-cb8c-4938-b19e-e37f6f99b165","Type":"ContainerStarted","Data":"3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab"} Oct 30 22:44:48.993254 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:48.992829 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:44:49.112821 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:49.112757 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" podStartSLOduration=3.166649306 podStartE2EDuration="4.112736702s" totalImagesPullingTime="946.087396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:49.108134213 +0000 UTC m=+3746.480760960" watchObservedRunningTime="2026-10-30 22:44:49.112736702 +0000 UTC m=+3746.485363450" Oct 30 22:44:55.005815 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:44:55.005779 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:45:00.192158 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.192112 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt"] Oct 30 22:45:00.196069 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.196043 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4"] Oct 30 22:45:00.196223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.196148 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" Oct 30 22:45:00.199237 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.199118 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" Oct 30 22:45:00.215286 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.215261 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r"] Oct 30 22:45:00.218378 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.218361 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" Oct 30 22:45:00.219702 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.219680 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc"] Oct 30 22:45:00.223204 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.223175 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" Oct 30 22:45:00.234505 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.234485 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl"] Oct 30 22:45:00.237191 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.237174 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" Oct 30 22:45:00.279691 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.279645 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt"] Oct 30 22:45:00.283979 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.283917 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4"] Oct 30 22:45:00.318321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.318288 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc"] Oct 30 22:45:00.329182 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.329147 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl"] Oct 30 22:45:00.329779 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.329755 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\"" Oct 30 22:45:00.344363 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344335 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8td8\" (UniqueName: \"kubernetes.io/projected/651c95b3-bdd2-4bcd-a299-73c43fe919dc-kube-api-access-h8td8\") pod \"maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r\" (UID: \"651c95b3-bdd2-4bcd-a299-73c43fe919dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" Oct 30 22:45:00.344489 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344372 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bffq7\" (UniqueName: \"kubernetes.io/projected/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-kube-api-access-bffq7\") pod \"maas-api-key-cleanup-29890005-z6gg4\" (UID: \"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" Oct 30 22:45:00.344489 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344397 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651c95b3-bdd2-4bcd-a299-73c43fe919dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r\" (UID: \"651c95b3-bdd2-4bcd-a299-73c43fe919dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" Oct 30 22:45:00.344489 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344442 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-z6gg4\" (UID: \"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" Oct 30 22:45:00.344489 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344458 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7mn\" (UniqueName: \"kubernetes.io/projected/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-kube-api-access-2j7mn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl\" (UID: \"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" Oct 30 22:45:00.344638 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344492 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8td\" (UniqueName: \"kubernetes.io/projected/29747294-37cb-465a-aa8f-b67d7a64c72a-kube-api-access-rj8td\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc\" (UID: \"29747294-37cb-465a-aa8f-b67d7a64c72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" Oct 30 22:45:00.344638 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344509 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl\" (UID: \"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" Oct 30 22:45:00.344638 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344542 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29747294-37cb-465a-aa8f-b67d7a64c72a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc\" (UID: \"29747294-37cb-465a-aa8f-b67d7a64c72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" Oct 30 22:45:00.344638 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344550 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r"] Oct 30 22:45:00.344775 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344640 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa5067-c0fe-4992-9d93-6841065c0348-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt\" (UID: \"3aaa5067-c0fe-4992-9d93-6841065c0348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" Oct 30 22:45:00.344775 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.344765 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84tp\" (UniqueName: \"kubernetes.io/projected/3aaa5067-c0fe-4992-9d93-6841065c0348-kube-api-access-w84tp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt\" (UID: \"3aaa5067-c0fe-4992-9d93-6841065c0348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" Oct 30 22:45:00.445557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.445484 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84tp\" (UniqueName: \"kubernetes.io/projected/3aaa5067-c0fe-4992-9d93-6841065c0348-kube-api-access-w84tp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt\" (UID: \"3aaa5067-c0fe-4992-9d93-6841065c0348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" Oct 30 22:45:00.445557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.445539 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8td8\" (UniqueName: \"kubernetes.io/projected/651c95b3-bdd2-4bcd-a299-73c43fe919dc-kube-api-access-h8td8\") pod \"maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r\" (UID: \"651c95b3-bdd2-4bcd-a299-73c43fe919dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" Oct 30 22:45:00.445785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.445558 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffq7\" (UniqueName: \"kubernetes.io/projected/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-kube-api-access-bffq7\") pod \"maas-api-key-cleanup-29890005-z6gg4\" (UID: \"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" Oct 30 22:45:00.445785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.445583 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651c95b3-bdd2-4bcd-a299-73c43fe919dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r\" (UID: \"651c95b3-bdd2-4bcd-a299-73c43fe919dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" Oct 30 22:45:00.445785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.445620 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-z6gg4\" (UID: \"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" Oct 30 22:45:00.445785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.445636 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7mn\" (UniqueName: \"kubernetes.io/projected/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-kube-api-access-2j7mn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl\" (UID: \"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" Oct 30 22:45:00.446028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.445787 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8td\" (UniqueName: \"kubernetes.io/projected/29747294-37cb-465a-aa8f-b67d7a64c72a-kube-api-access-rj8td\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc\" (UID: \"29747294-37cb-465a-aa8f-b67d7a64c72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" Oct 30 22:45:00.446028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.445839 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl\" (UID: \"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" Oct 30 22:45:00.446028 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.445888 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29747294-37cb-465a-aa8f-b67d7a64c72a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc\" (UID: \"29747294-37cb-465a-aa8f-b67d7a64c72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" Oct 30 22:45:00.446177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.446047 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa5067-c0fe-4992-9d93-6841065c0348-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt\" (UID: \"3aaa5067-c0fe-4992-9d93-6841065c0348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" Oct 30 22:45:00.446518 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.446496 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651c95b3-bdd2-4bcd-a299-73c43fe919dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r\" (UID: \"651c95b3-bdd2-4bcd-a299-73c43fe919dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" Oct 30 22:45:00.446557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.446530 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29747294-37cb-465a-aa8f-b67d7a64c72a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc\" (UID: \"29747294-37cb-465a-aa8f-b67d7a64c72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" Oct 30 22:45:00.446557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.446505 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-z6gg4\" (UID: \"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" Oct 30 22:45:00.446557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.446550 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl\" (UID: \"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" Oct 30 22:45:00.446730 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.446713 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa5067-c0fe-4992-9d93-6841065c0348-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt\" (UID: \"3aaa5067-c0fe-4992-9d93-6841065c0348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" Oct 30 22:45:00.503424 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.503344 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8td8\" (UniqueName: \"kubernetes.io/projected/651c95b3-bdd2-4bcd-a299-73c43fe919dc-kube-api-access-h8td8\") pod \"maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r\" (UID: \"651c95b3-bdd2-4bcd-a299-73c43fe919dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" Oct 30 22:45:00.503424 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.503344 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84tp\" (UniqueName: \"kubernetes.io/projected/3aaa5067-c0fe-4992-9d93-6841065c0348-kube-api-access-w84tp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt\" (UID: \"3aaa5067-c0fe-4992-9d93-6841065c0348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" Oct 30 22:45:00.503661 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.503552 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8td\" (UniqueName: \"kubernetes.io/projected/29747294-37cb-465a-aa8f-b67d7a64c72a-kube-api-access-rj8td\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc\" (UID: \"29747294-37cb-465a-aa8f-b67d7a64c72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" Oct 30 22:45:00.503781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.503760 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bffq7\" (UniqueName: \"kubernetes.io/projected/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-kube-api-access-bffq7\") pod \"maas-api-key-cleanup-29890005-z6gg4\" (UID: \"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" Oct 30 22:45:00.504423 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.504396 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7mn\" (UniqueName: \"kubernetes.io/projected/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-kube-api-access-2j7mn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl\" (UID: \"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" Oct 30 22:45:00.511320 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.511301 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" Oct 30 22:45:00.518723 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.518705 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" Oct 30 22:45:00.531190 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.531166 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" Oct 30 22:45:00.537986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.537962 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" Oct 30 22:45:00.547847 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.547824 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" Oct 30 22:45:00.919164 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.919133 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt"] Oct 30 22:45:00.919791 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:45:00.919767 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aaa5067_c0fe_4992_9d93_6841065c0348.slice/crio-cbb0f8d214e7c596bfa7f3183099f2c7fd9c16a3d4cab2a364445044dab88e71 WatchSource:0}: Error finding container cbb0f8d214e7c596bfa7f3183099f2c7fd9c16a3d4cab2a364445044dab88e71: Status 404 returned error can't find the container with id cbb0f8d214e7c596bfa7f3183099f2c7fd9c16a3d4cab2a364445044dab88e71 Oct 30 22:45:00.969177 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.969150 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r"] Oct 30 22:45:00.974402 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:45:00.974379 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651c95b3_bdd2_4bcd_a299_73c43fe919dc.slice/crio-4e7e2be53c2aa90ac25412787773dcf7c310bd762629f2424462604bd356d8e6 WatchSource:0}: Error finding container 4e7e2be53c2aa90ac25412787773dcf7c310bd762629f2424462604bd356d8e6: Status 404 returned error can't find the container with id 4e7e2be53c2aa90ac25412787773dcf7c310bd762629f2424462604bd356d8e6 Oct 30 22:45:00.977048 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:45:00.977009 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3e3b5b_0359_4d3d_b4dc_e03597c9c89f.slice/crio-ed24b2361d8c3142929fb2e3a61e0b53e874cbd68519b7a67672d28c74e10e1f WatchSource:0}: Error finding container ed24b2361d8c3142929fb2e3a61e0b53e874cbd68519b7a67672d28c74e10e1f: Status 404 returned error can't find the container with id ed24b2361d8c3142929fb2e3a61e0b53e874cbd68519b7a67672d28c74e10e1f Oct 30 22:45:00.977163 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.977096 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4"] Oct 30 22:45:00.989599 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:00.989576 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc"] Oct 30 22:45:00.995863 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:45:00.995838 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29747294_37cb_465a_aa8f_b67d7a64c72a.slice/crio-a8bcdbd61a13384017bd56aea3040ad4c4546405a032bf48963041070ccdef5c WatchSource:0}: Error finding container a8bcdbd61a13384017bd56aea3040ad4c4546405a032bf48963041070ccdef5c: Status 404 returned error can't find the container with id a8bcdbd61a13384017bd56aea3040ad4c4546405a032bf48963041070ccdef5c Oct 30 22:45:01.028136 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:01.028111 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl"] Oct 30 22:45:01.028721 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:45:01.028697 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae92f461_f0cc_4bdb_852c_b7d8d73aa4c3.slice/crio-29c29c6a08c5f8141ea31292870f229eb411a086bc467d181d6a53c2299ef280 WatchSource:0}: Error finding container 29c29c6a08c5f8141ea31292870f229eb411a086bc467d181d6a53c2299ef280: Status 404 returned error can't find the container with id 29c29c6a08c5f8141ea31292870f229eb411a086bc467d181d6a53c2299ef280 Oct 30 22:45:01.074503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:01.074438 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" event={"ID":"3aaa5067-c0fe-4992-9d93-6841065c0348","Type":"ContainerStarted","Data":"cbb0f8d214e7c596bfa7f3183099f2c7fd9c16a3d4cab2a364445044dab88e71"} Oct 30 22:45:01.079963 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:01.076015 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" event={"ID":"29747294-37cb-465a-aa8f-b67d7a64c72a","Type":"ContainerStarted","Data":"a8bcdbd61a13384017bd56aea3040ad4c4546405a032bf48963041070ccdef5c"} Oct 30 22:45:01.082110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:01.082081 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" event={"ID":"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f","Type":"ContainerStarted","Data":"ed24b2361d8c3142929fb2e3a61e0b53e874cbd68519b7a67672d28c74e10e1f"} Oct 30 22:45:01.083537 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:01.083507 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" event={"ID":"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3","Type":"ContainerStarted","Data":"29c29c6a08c5f8141ea31292870f229eb411a086bc467d181d6a53c2299ef280"} Oct 30 22:45:01.084954 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:01.084921 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" event={"ID":"651c95b3-bdd2-4bcd-a299-73c43fe919dc","Type":"ContainerStarted","Data":"4e7e2be53c2aa90ac25412787773dcf7c310bd762629f2424462604bd356d8e6"} Oct 30 22:45:02.093340 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.093306 4859 generic.go:356] "Generic (PLEG): container finished" podID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerID="cd52d623b7cb319cc8660fc0e7dcf5f0825242619e0dc927b139ee1b987e15f6" exitCode=60 Oct 30 22:45:02.093722 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.093382 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" event={"ID":"651c95b3-bdd2-4bcd-a299-73c43fe919dc","Type":"ContainerDied","Data":"cd52d623b7cb319cc8660fc0e7dcf5f0825242619e0dc927b139ee1b987e15f6"} Oct 30 22:45:02.093722 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.093662 4859 scope.go:122] "RemoveContainer" containerID="cd52d623b7cb319cc8660fc0e7dcf5f0825242619e0dc927b139ee1b987e15f6" Oct 30 22:45:02.096119 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.095735 4859 generic.go:356] "Generic (PLEG): container finished" podID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerID="eabc651bacaa292445ba57cd85c846ad98f2290544102629b791d8495c3cea7c" exitCode=60 Oct 30 22:45:02.096119 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.095824 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" event={"ID":"3aaa5067-c0fe-4992-9d93-6841065c0348","Type":"ContainerDied","Data":"eabc651bacaa292445ba57cd85c846ad98f2290544102629b791d8495c3cea7c"} Oct 30 22:45:02.096250 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.096128 4859 scope.go:122] "RemoveContainer" containerID="eabc651bacaa292445ba57cd85c846ad98f2290544102629b791d8495c3cea7c" Oct 30 22:45:02.105277 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.105253 4859 generic.go:356] "Generic (PLEG): container finished" podID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerID="d68a60f33615dbfb7693f5306c965263c039ff42277f607f27689852239345e1" exitCode=60 Oct 30 22:45:02.105402 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.105336 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" event={"ID":"29747294-37cb-465a-aa8f-b67d7a64c72a","Type":"ContainerDied","Data":"d68a60f33615dbfb7693f5306c965263c039ff42277f607f27689852239345e1"} Oct 30 22:45:02.105594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.105580 4859 scope.go:122] "RemoveContainer" containerID="d68a60f33615dbfb7693f5306c965263c039ff42277f607f27689852239345e1" Oct 30 22:45:02.107533 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.107507 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" event={"ID":"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f","Type":"ContainerStarted","Data":"9c53ccd6374ec921a90b32e33ac3b2560db34279d9ca6559948242b7991bd427"} Oct 30 22:45:02.111032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.110643 4859 generic.go:356] "Generic (PLEG): container finished" podID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerID="31301a5cb5d67f45d86b8657d329ec90a2f935cd512f87678a7b941cfe588cbe" exitCode=60 Oct 30 22:45:02.111032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.110694 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" event={"ID":"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3","Type":"ContainerDied","Data":"31301a5cb5d67f45d86b8657d329ec90a2f935cd512f87678a7b941cfe588cbe"} Oct 30 22:45:02.111032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.110899 4859 scope.go:122] "RemoveContainer" containerID="31301a5cb5d67f45d86b8657d329ec90a2f935cd512f87678a7b941cfe588cbe" Oct 30 22:45:02.336828 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:02.336778 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" podStartSLOduration=2.336765856 podStartE2EDuration="2.336765856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.335264065 +0000 UTC m=+3759.707890809" watchObservedRunningTime="2026-10-30 22:45:02.336765856 +0000 UTC m=+3759.709392602" Oct 30 22:45:03.120388 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.120270 4859 generic.go:356] "Generic (PLEG): container finished" podID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerID="6cd85754828648383868bf6af8e17af594885e3d1e0e41be6224da13e4d32b04" exitCode=60 Oct 30 22:45:03.120388 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.120355 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" event={"ID":"651c95b3-bdd2-4bcd-a299-73c43fe919dc","Type":"ContainerDied","Data":"6cd85754828648383868bf6af8e17af594885e3d1e0e41be6224da13e4d32b04"} Oct 30 22:45:03.120783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.120415 4859 scope.go:122] "RemoveContainer" containerID="cd52d623b7cb319cc8660fc0e7dcf5f0825242619e0dc927b139ee1b987e15f6" Oct 30 22:45:03.120783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.120714 4859 scope.go:122] "RemoveContainer" containerID="6cd85754828648383868bf6af8e17af594885e3d1e0e41be6224da13e4d32b04" Oct 30 22:45:03.121096 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:03.121019 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r_odh-ai-gateway-infra(651c95b3-bdd2-4bcd-a299-73c43fe919dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" Oct 30 22:45:03.123103 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.123079 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" event={"ID":"3aaa5067-c0fe-4992-9d93-6841065c0348","Type":"ContainerStarted","Data":"0c597f9f844857396725cfbf2056d4b34bb6eacc476b134dceca22fe4fc876e7"} Oct 30 22:45:03.125052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.125026 4859 generic.go:356] "Generic (PLEG): container finished" podID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerID="8794f768e7baac267acf82c49d49401b5901f620514d70111eafb519be4c24ad" exitCode=60 Oct 30 22:45:03.125150 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.125106 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" event={"ID":"29747294-37cb-465a-aa8f-b67d7a64c72a","Type":"ContainerDied","Data":"8794f768e7baac267acf82c49d49401b5901f620514d70111eafb519be4c24ad"} Oct 30 22:45:03.125371 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.125353 4859 scope.go:122] "RemoveContainer" containerID="8794f768e7baac267acf82c49d49401b5901f620514d70111eafb519be4c24ad" Oct 30 22:45:03.125659 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:03.125637 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc_odh-ai-gateway-infra(29747294-37cb-465a-aa8f-b67d7a64c72a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" Oct 30 22:45:03.126728 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.126708 4859 generic.go:356] "Generic (PLEG): container finished" podID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerID="9c53ccd6374ec921a90b32e33ac3b2560db34279d9ca6559948242b7991bd427" exitCode=60 Oct 30 22:45:03.126817 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.126785 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" event={"ID":"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f","Type":"ContainerDied","Data":"9c53ccd6374ec921a90b32e33ac3b2560db34279d9ca6559948242b7991bd427"} Oct 30 22:45:03.126991 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.126974 4859 scope.go:122] "RemoveContainer" containerID="9c53ccd6374ec921a90b32e33ac3b2560db34279d9ca6559948242b7991bd427" Oct 30 22:45:03.128759 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.128740 4859 generic.go:356] "Generic (PLEG): container finished" podID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerID="179c0bfdb972c1518faff7dd471ea26f70e894191c37ab8351f13dbffec5c6b2" exitCode=60 Oct 30 22:45:03.128844 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.128791 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" event={"ID":"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3","Type":"ContainerDied","Data":"179c0bfdb972c1518faff7dd471ea26f70e894191c37ab8351f13dbffec5c6b2"} Oct 30 22:45:03.129048 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.129031 4859 scope.go:122] "RemoveContainer" containerID="179c0bfdb972c1518faff7dd471ea26f70e894191c37ab8351f13dbffec5c6b2" Oct 30 22:45:03.130572 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:03.130554 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl_odh-ai-gateway-infra(ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" Oct 30 22:45:03.239376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.239235 4859 scope.go:122] "RemoveContainer" containerID="d68a60f33615dbfb7693f5306c965263c039ff42277f607f27689852239345e1" Oct 30 22:45:03.252348 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.252327 4859 scope.go:122] "RemoveContainer" containerID="31301a5cb5d67f45d86b8657d329ec90a2f935cd512f87678a7b941cfe588cbe" Oct 30 22:45:03.575805 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:03.575702 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" podStartSLOduration=3.575688431 podStartE2EDuration="3.575688431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.573590203 +0000 UTC m=+3760.946216950" watchObservedRunningTime="2026-10-30 22:45:03.575688431 +0000 UTC m=+3760.948315177" Oct 30 22:45:04.137264 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.137239 4859 scope.go:122] "RemoveContainer" containerID="179c0bfdb972c1518faff7dd471ea26f70e894191c37ab8351f13dbffec5c6b2" Oct 30 22:45:04.137623 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:04.137528 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl_odh-ai-gateway-infra(ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" Oct 30 22:45:04.138907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.138887 4859 scope.go:122] "RemoveContainer" containerID="6cd85754828648383868bf6af8e17af594885e3d1e0e41be6224da13e4d32b04" Oct 30 22:45:04.139712 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:04.139686 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r_odh-ai-gateway-infra(651c95b3-bdd2-4bcd-a299-73c43fe919dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" Oct 30 22:45:04.140602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.140578 4859 generic.go:356] "Generic (PLEG): container finished" podID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerID="0c597f9f844857396725cfbf2056d4b34bb6eacc476b134dceca22fe4fc876e7" exitCode=60 Oct 30 22:45:04.140709 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.140656 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" event={"ID":"3aaa5067-c0fe-4992-9d93-6841065c0348","Type":"ContainerDied","Data":"0c597f9f844857396725cfbf2056d4b34bb6eacc476b134dceca22fe4fc876e7"} Oct 30 22:45:04.140779 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.140715 4859 scope.go:122] "RemoveContainer" containerID="eabc651bacaa292445ba57cd85c846ad98f2290544102629b791d8495c3cea7c" Oct 30 22:45:04.140890 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.140872 4859 scope.go:122] "RemoveContainer" containerID="0c597f9f844857396725cfbf2056d4b34bb6eacc476b134dceca22fe4fc876e7" Oct 30 22:45:04.141148 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:04.141124 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt_odh-ai-gateway-infra(3aaa5067-c0fe-4992-9d93-6841065c0348)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" Oct 30 22:45:04.142897 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.142881 4859 scope.go:122] "RemoveContainer" containerID="8794f768e7baac267acf82c49d49401b5901f620514d70111eafb519be4c24ad" Oct 30 22:45:04.143269 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:04.143248 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc_odh-ai-gateway-infra(29747294-37cb-465a-aa8f-b67d7a64c72a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" Oct 30 22:45:04.144576 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.144556 4859 generic.go:356] "Generic (PLEG): container finished" podID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerID="becc587750107cc84a6b8a89d086faf8b5604f0f4fdf451b2ae858bea1f5f3ee" exitCode=60 Oct 30 22:45:04.144673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.144616 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" event={"ID":"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f","Type":"ContainerDied","Data":"becc587750107cc84a6b8a89d086faf8b5604f0f4fdf451b2ae858bea1f5f3ee"} Oct 30 22:45:04.144871 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.144850 4859 scope.go:122] "RemoveContainer" containerID="becc587750107cc84a6b8a89d086faf8b5604f0f4fdf451b2ae858bea1f5f3ee" Oct 30 22:45:04.145088 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:04.145070 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-z6gg4_odh-ai-gateway-infra(7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" Oct 30 22:45:04.152609 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:04.152586 4859 scope.go:122] "RemoveContainer" containerID="9c53ccd6374ec921a90b32e33ac3b2560db34279d9ca6559948242b7991bd427" Oct 30 22:45:05.153699 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:05.153669 4859 scope.go:122] "RemoveContainer" containerID="becc587750107cc84a6b8a89d086faf8b5604f0f4fdf451b2ae858bea1f5f3ee" Oct 30 22:45:05.154196 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:05.154032 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-z6gg4_odh-ai-gateway-infra(7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" Oct 30 22:45:05.155708 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:05.155687 4859 scope.go:122] "RemoveContainer" containerID="0c597f9f844857396725cfbf2056d4b34bb6eacc476b134dceca22fe4fc876e7" Oct 30 22:45:05.155929 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:05.155910 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt_odh-ai-gateway-infra(3aaa5067-c0fe-4992-9d93-6841065c0348)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" Oct 30 22:45:06.756694 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:06.756647 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:45:08.775718 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.775683 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck"] Oct 30 22:45:08.780883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.780865 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.883266 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.883221 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-podinfo\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.883513 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.883493 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-envoy\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.883644 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.883628 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-certs\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.883793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.883777 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d35c973-cc54-443b-b0a4-add6d36a409c-istiod-ca-cert\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.883916 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.883893 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-data\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.884068 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.884052 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-token\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.884224 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.884193 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-credential-socket\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.884504 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.884454 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-socket\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.884687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.884670 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hcd\" (UniqueName: \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-kube-api-access-86hcd\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.908706 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.908608 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck"] Oct 30 22:45:08.941980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.939405 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-69058c-openshift-default-dockercfg-fwc5l\"" Oct 30 22:45:08.985925 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.985893 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-socket\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986124 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.985970 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hcd\" (UniqueName: \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-kube-api-access-86hcd\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986124 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986025 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-podinfo\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986249 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986138 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-envoy\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986249 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986189 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-certs\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986351 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986261 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d35c973-cc54-443b-b0a4-add6d36a409c-istiod-ca-cert\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986351 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986292 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-data\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986351 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986326 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-socket\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986504 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986418 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-token\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986504 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986481 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-credential-socket\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986543 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-certs\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986557 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-data\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986840 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986812 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-credential-socket\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.986992 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.986928 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d35c973-cc54-443b-b0a4-add6d36a409c-istiod-ca-cert\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.988547 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.988527 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-envoy\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:08.988787 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:08.988766 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-podinfo\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:09.042962 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:09.042891 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-token\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:09.042962 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:09.042924 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hcd\" (UniqueName: \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-kube-api-access-86hcd\") pod \"e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:09.094362 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:09.094329 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:09.522658 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:09.522626 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck"] Oct 30 22:45:09.523854 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:45:09.523827 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d35c973_cc54_443b_b0a4_add6d36a409c.slice/crio-e789601483b1612ad34077c43b56e58d7c739eb0ab1ddf94d2b22cc14d612b74 WatchSource:0}: Error finding container e789601483b1612ad34077c43b56e58d7c739eb0ab1ddf94d2b22cc14d612b74: Status 404 returned error can't find the container with id e789601483b1612ad34077c43b56e58d7c739eb0ab1ddf94d2b22cc14d612b74 Oct 30 22:45:09.526408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:09.526376 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:45:09.526471 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:09.526439 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:45:09.526510 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:09.526472 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:45:09.877223 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:09.877185 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:45:10.211645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:10.211569 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" event={"ID":"6d35c973-cc54-443b-b0a4-add6d36a409c","Type":"ContainerStarted","Data":"ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad"} Oct 30 22:45:10.211645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:10.211642 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" event={"ID":"6d35c973-cc54-443b-b0a4-add6d36a409c","Type":"ContainerStarted","Data":"e789601483b1612ad34077c43b56e58d7c739eb0ab1ddf94d2b22cc14d612b74"} Oct 30 22:45:10.330378 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:10.330332 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" podStartSLOduration=2.330319752 podStartE2EDuration="2.330319752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:10.327906352 +0000 UTC m=+3767.700533098" watchObservedRunningTime="2026-10-30 22:45:10.330319752 +0000 UTC m=+3767.702946560" Oct 30 22:45:11.094600 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:11.094567 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:11.100811 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:11.100780 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:11.232502 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:11.232465 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:11.233931 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:11.233912 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:45:14.734067 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:14.734036 4859 scope.go:122] "RemoveContainer" containerID="0c597f9f844857396725cfbf2056d4b34bb6eacc476b134dceca22fe4fc876e7" Oct 30 22:45:14.734441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:14.734154 4859 scope.go:122] "RemoveContainer" containerID="8794f768e7baac267acf82c49d49401b5901f620514d70111eafb519be4c24ad" Oct 30 22:45:15.263246 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:15.263193 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" event={"ID":"3aaa5067-c0fe-4992-9d93-6841065c0348","Type":"ContainerStarted","Data":"c7daec348a75693a94cc3352ff8dba959e0cb00899c7a0f575975f9ad93957f0"} Oct 30 22:45:15.265634 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:15.265599 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" event={"ID":"29747294-37cb-465a-aa8f-b67d7a64c72a","Type":"ContainerStarted","Data":"cd3c53864971592faa5841687230eaedc09230cac24eb51e6e3d1f408462746a"} Oct 30 22:45:15.494317 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:15.494264 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" podStartSLOduration=15.494250641 podStartE2EDuration="15.494250641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:15.491737739 +0000 UTC m=+3772.864364485" watchObservedRunningTime="2026-10-30 22:45:15.494250641 +0000 UTC m=+3772.866877423" Oct 30 22:45:15.734275 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:15.734250 4859 scope.go:122] "RemoveContainer" containerID="becc587750107cc84a6b8a89d086faf8b5604f0f4fdf451b2ae858bea1f5f3ee" Oct 30 22:45:15.734669 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:15.734444 4859 scope.go:122] "RemoveContainer" containerID="6cd85754828648383868bf6af8e17af594885e3d1e0e41be6224da13e4d32b04" Oct 30 22:45:16.277260 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.277220 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" event={"ID":"651c95b3-bdd2-4bcd-a299-73c43fe919dc","Type":"ContainerStarted","Data":"cbff1f1995640a299d6fa306181612d04b17803e51f734d64038412c66ca064f"} Oct 30 22:45:16.279296 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.279266 4859 generic.go:356] "Generic (PLEG): container finished" podID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerID="c7daec348a75693a94cc3352ff8dba959e0cb00899c7a0f575975f9ad93957f0" exitCode=60 Oct 30 22:45:16.279432 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.279345 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" event={"ID":"3aaa5067-c0fe-4992-9d93-6841065c0348","Type":"ContainerDied","Data":"c7daec348a75693a94cc3352ff8dba959e0cb00899c7a0f575975f9ad93957f0"} Oct 30 22:45:16.279432 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.279401 4859 scope.go:122] "RemoveContainer" containerID="0c597f9f844857396725cfbf2056d4b34bb6eacc476b134dceca22fe4fc876e7" Oct 30 22:45:16.279756 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.279734 4859 scope.go:122] "RemoveContainer" containerID="c7daec348a75693a94cc3352ff8dba959e0cb00899c7a0f575975f9ad93957f0" Oct 30 22:45:16.280124 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:16.280099 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt_odh-ai-gateway-infra(3aaa5067-c0fe-4992-9d93-6841065c0348)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" Oct 30 22:45:16.281836 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.281813 4859 generic.go:356] "Generic (PLEG): container finished" podID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerID="cd3c53864971592faa5841687230eaedc09230cac24eb51e6e3d1f408462746a" exitCode=60 Oct 30 22:45:16.281925 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.281878 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" event={"ID":"29747294-37cb-465a-aa8f-b67d7a64c72a","Type":"ContainerDied","Data":"cd3c53864971592faa5841687230eaedc09230cac24eb51e6e3d1f408462746a"} Oct 30 22:45:16.282225 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.282207 4859 scope.go:122] "RemoveContainer" containerID="cd3c53864971592faa5841687230eaedc09230cac24eb51e6e3d1f408462746a" Oct 30 22:45:16.282438 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:16.282416 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc_odh-ai-gateway-infra(29747294-37cb-465a-aa8f-b67d7a64c72a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" Oct 30 22:45:16.283985 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.283964 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" event={"ID":"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f","Type":"ContainerStarted","Data":"55217cbdee63beff22a618007d3da3d23e0d082480eb15a18f38c196c8d134d9"} Oct 30 22:45:16.295725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.295699 4859 scope.go:122] "RemoveContainer" containerID="8794f768e7baac267acf82c49d49401b5901f620514d70111eafb519be4c24ad" Oct 30 22:45:16.388255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.388226 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt"] Oct 30 22:45:16.434893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.434844 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" podStartSLOduration=16.434831802 podStartE2EDuration="16.434831802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:16.394837601 +0000 UTC m=+3773.767464346" watchObservedRunningTime="2026-10-30 22:45:16.434831802 +0000 UTC m=+3773.807458589" Oct 30 22:45:16.512305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:16.512270 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc"] Oct 30 22:45:17.294484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.294446 4859 generic.go:356] "Generic (PLEG): container finished" podID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerID="cbff1f1995640a299d6fa306181612d04b17803e51f734d64038412c66ca064f" exitCode=60 Oct 30 22:45:17.295025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.294526 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" event={"ID":"651c95b3-bdd2-4bcd-a299-73c43fe919dc","Type":"ContainerDied","Data":"cbff1f1995640a299d6fa306181612d04b17803e51f734d64038412c66ca064f"} Oct 30 22:45:17.295025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.294609 4859 scope.go:122] "RemoveContainer" containerID="6cd85754828648383868bf6af8e17af594885e3d1e0e41be6224da13e4d32b04" Oct 30 22:45:17.295518 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.295496 4859 scope.go:122] "RemoveContainer" containerID="cbff1f1995640a299d6fa306181612d04b17803e51f734d64038412c66ca064f" Oct 30 22:45:17.295922 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:17.295857 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r_odh-ai-gateway-infra(651c95b3-bdd2-4bcd-a299-73c43fe919dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" Oct 30 22:45:17.307396 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.306509 4859 generic.go:356] "Generic (PLEG): container finished" podID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerID="55217cbdee63beff22a618007d3da3d23e0d082480eb15a18f38c196c8d134d9" exitCode=60 Oct 30 22:45:17.307396 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.306622 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" event={"ID":"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f","Type":"ContainerDied","Data":"55217cbdee63beff22a618007d3da3d23e0d082480eb15a18f38c196c8d134d9"} Oct 30 22:45:17.307396 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.306995 4859 scope.go:122] "RemoveContainer" containerID="55217cbdee63beff22a618007d3da3d23e0d082480eb15a18f38c196c8d134d9" Oct 30 22:45:17.307396 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:17.307293 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-z6gg4_odh-ai-gateway-infra(7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" Oct 30 22:45:17.310253 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.310216 4859 scope.go:122] "RemoveContainer" containerID="becc587750107cc84a6b8a89d086faf8b5604f0f4fdf451b2ae858bea1f5f3ee" Oct 30 22:45:17.408685 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.408647 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r"] Oct 30 22:45:17.483269 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.483239 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" Oct 30 22:45:17.528374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.528351 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" Oct 30 22:45:17.615966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.613410 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aaa5067-c0fe-4992-9d93-6841065c0348-kube-api-access-w84tp\" (UniqueName: \"kubernetes.io/projected/3aaa5067-c0fe-4992-9d93-6841065c0348-kube-api-access-w84tp\") pod \"3aaa5067-c0fe-4992-9d93-6841065c0348\" (UID: \"3aaa5067-c0fe-4992-9d93-6841065c0348\") " Oct 30 22:45:17.615966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.613506 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29747294-37cb-465a-aa8f-b67d7a64c72a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29747294-37cb-465a-aa8f-b67d7a64c72a-maas-api-service-ca\") pod \"29747294-37cb-465a-aa8f-b67d7a64c72a\" (UID: \"29747294-37cb-465a-aa8f-b67d7a64c72a\") " Oct 30 22:45:17.615966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.613704 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29747294-37cb-465a-aa8f-b67d7a64c72a-kube-api-access-rj8td\" (UniqueName: \"kubernetes.io/projected/29747294-37cb-465a-aa8f-b67d7a64c72a-kube-api-access-rj8td\") pod \"29747294-37cb-465a-aa8f-b67d7a64c72a\" (UID: \"29747294-37cb-465a-aa8f-b67d7a64c72a\") " Oct 30 22:45:17.615966 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.613772 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aaa5067-c0fe-4992-9d93-6841065c0348-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa5067-c0fe-4992-9d93-6841065c0348-maas-api-service-ca\") pod \"3aaa5067-c0fe-4992-9d93-6841065c0348\" (UID: \"3aaa5067-c0fe-4992-9d93-6841065c0348\") " Oct 30 22:45:17.626758 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.625599 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aaa5067-c0fe-4992-9d93-6841065c0348-maas-api-service-ca" pod "3aaa5067-c0fe-4992-9d93-6841065c0348" (UID: "3aaa5067-c0fe-4992-9d93-6841065c0348"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:17.629367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.627221 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29747294-37cb-465a-aa8f-b67d7a64c72a-maas-api-service-ca" pod "29747294-37cb-465a-aa8f-b67d7a64c72a" (UID: "29747294-37cb-465a-aa8f-b67d7a64c72a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:17.647505 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.647457 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aaa5067-c0fe-4992-9d93-6841065c0348-kube-api-access-w84tp" pod "3aaa5067-c0fe-4992-9d93-6841065c0348" (UID: "3aaa5067-c0fe-4992-9d93-6841065c0348"). InnerVolumeSpecName "kube-api-access-w84tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:17.647640 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.647511 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29747294-37cb-465a-aa8f-b67d7a64c72a-kube-api-access-rj8td" pod "29747294-37cb-465a-aa8f-b67d7a64c72a" (UID: "29747294-37cb-465a-aa8f-b67d7a64c72a"). InnerVolumeSpecName "kube-api-access-rj8td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:17.659229 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.659202 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4"] Oct 30 22:45:17.717438 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.717405 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w84tp\" (UniqueName: \"kubernetes.io/projected/3aaa5067-c0fe-4992-9d93-6841065c0348-kube-api-access-w84tp\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:17.717438 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.717437 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29747294-37cb-465a-aa8f-b67d7a64c72a-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:17.717603 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.717454 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj8td\" (UniqueName: \"kubernetes.io/projected/29747294-37cb-465a-aa8f-b67d7a64c72a-kube-api-access-rj8td\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:17.717603 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.717468 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa5067-c0fe-4992-9d93-6841065c0348-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:17.734044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:17.734015 4859 scope.go:122] "RemoveContainer" containerID="179c0bfdb972c1518faff7dd471ea26f70e894191c37ab8351f13dbffec5c6b2" Oct 30 22:45:18.319323 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.319285 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" event={"ID":"3aaa5067-c0fe-4992-9d93-6841065c0348","Type":"ContainerDied","Data":"cbb0f8d214e7c596bfa7f3183099f2c7fd9c16a3d4cab2a364445044dab88e71"} Oct 30 22:45:18.319699 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.319356 4859 scope.go:122] "RemoveContainer" containerID="c7daec348a75693a94cc3352ff8dba959e0cb00899c7a0f575975f9ad93957f0" Oct 30 22:45:18.319699 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.319493 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt" Oct 30 22:45:18.345999 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.330896 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" event={"ID":"29747294-37cb-465a-aa8f-b67d7a64c72a","Type":"ContainerDied","Data":"a8bcdbd61a13384017bd56aea3040ad4c4546405a032bf48963041070ccdef5c"} Oct 30 22:45:18.345999 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.331053 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc" Oct 30 22:45:18.345999 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.342621 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" event={"ID":"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3","Type":"ContainerStarted","Data":"a236181199665d1736b0868c6ef51dbd3eb1eaf1fa337ae6ac57404d0a74c758"} Oct 30 22:45:18.361128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.358171 4859 scope.go:122] "RemoveContainer" containerID="cd3c53864971592faa5841687230eaedc09230cac24eb51e6e3d1f408462746a" Oct 30 22:45:18.539653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.539629 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" Oct 30 22:45:18.550411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.550361 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" podStartSLOduration=18.550343516 podStartE2EDuration="18.550343516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:18.549432207 +0000 UTC m=+3775.922058953" watchObservedRunningTime="2026-10-30 22:45:18.550343516 +0000 UTC m=+3775.922970263" Oct 30 22:45:18.581880 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.581824 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" Oct 30 22:45:18.649843 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.649811 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-maas-api-service-ca\") pod \"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f\" (UID: \"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f\") " Oct 30 22:45:18.649996 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.649864 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651c95b3-bdd2-4bcd-a299-73c43fe919dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651c95b3-bdd2-4bcd-a299-73c43fe919dc-maas-api-service-ca\") pod \"651c95b3-bdd2-4bcd-a299-73c43fe919dc\" (UID: \"651c95b3-bdd2-4bcd-a299-73c43fe919dc\") " Oct 30 22:45:18.649996 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.649968 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651c95b3-bdd2-4bcd-a299-73c43fe919dc-kube-api-access-h8td8\" (UniqueName: \"kubernetes.io/projected/651c95b3-bdd2-4bcd-a299-73c43fe919dc-kube-api-access-h8td8\") pod \"651c95b3-bdd2-4bcd-a299-73c43fe919dc\" (UID: \"651c95b3-bdd2-4bcd-a299-73c43fe919dc\") " Oct 30 22:45:18.650129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.650041 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-kube-api-access-bffq7\" (UniqueName: \"kubernetes.io/projected/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-kube-api-access-bffq7\") pod \"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f\" (UID: \"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f\") " Oct 30 22:45:18.650234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.650207 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-maas-api-service-ca" pod "7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" (UID: "7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:18.650289 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.650256 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651c95b3-bdd2-4bcd-a299-73c43fe919dc-maas-api-service-ca" pod "651c95b3-bdd2-4bcd-a299-73c43fe919dc" (UID: "651c95b3-bdd2-4bcd-a299-73c43fe919dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:18.650474 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.650457 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.650535 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.650479 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651c95b3-bdd2-4bcd-a299-73c43fe919dc-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.651964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.651929 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651c95b3-bdd2-4bcd-a299-73c43fe919dc-kube-api-access-h8td8" pod "651c95b3-bdd2-4bcd-a299-73c43fe919dc" (UID: "651c95b3-bdd2-4bcd-a299-73c43fe919dc"). InnerVolumeSpecName "kube-api-access-h8td8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:18.652029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.651965 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-kube-api-access-bffq7" pod "7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" (UID: "7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f"). InnerVolumeSpecName "kube-api-access-bffq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:18.666114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.666050 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc"] Oct 30 22:45:18.670442 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.670409 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29890005-qqdwc"] Oct 30 22:45:18.742602 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.742574 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" path="/var/lib/kubelet/pods/29747294-37cb-465a-aa8f-b67d7a64c72a/volumes" Oct 30 22:45:18.751581 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.751564 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8td8\" (UniqueName: \"kubernetes.io/projected/651c95b3-bdd2-4bcd-a299-73c43fe919dc-kube-api-access-h8td8\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.751660 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.751584 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bffq7\" (UniqueName: \"kubernetes.io/projected/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f-kube-api-access-bffq7\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.890497 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.890471 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt"] Oct 30 22:45:18.897052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:18.897026 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f7654-29890005-5wpwt"] Oct 30 22:45:19.354615 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.354584 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" Oct 30 22:45:19.354615 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.354605 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4" event={"ID":"7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f","Type":"ContainerDied","Data":"ed24b2361d8c3142929fb2e3a61e0b53e874cbd68519b7a67672d28c74e10e1f"} Oct 30 22:45:19.355222 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.354687 4859 scope.go:122] "RemoveContainer" containerID="55217cbdee63beff22a618007d3da3d23e0d082480eb15a18f38c196c8d134d9" Oct 30 22:45:19.357251 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.357222 4859 generic.go:356] "Generic (PLEG): container finished" podID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerID="a236181199665d1736b0868c6ef51dbd3eb1eaf1fa337ae6ac57404d0a74c758" exitCode=60 Oct 30 22:45:19.357364 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.357262 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" event={"ID":"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3","Type":"ContainerDied","Data":"a236181199665d1736b0868c6ef51dbd3eb1eaf1fa337ae6ac57404d0a74c758"} Oct 30 22:45:19.357692 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.357671 4859 scope.go:122] "RemoveContainer" containerID="a236181199665d1736b0868c6ef51dbd3eb1eaf1fa337ae6ac57404d0a74c758" Oct 30 22:45:19.358037 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:19.358007 4859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl_odh-ai-gateway-infra(ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" Oct 30 22:45:19.358775 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.358758 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" Oct 30 22:45:19.358851 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.358789 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r" event={"ID":"651c95b3-bdd2-4bcd-a299-73c43fe919dc","Type":"ContainerDied","Data":"4e7e2be53c2aa90ac25412787773dcf7c310bd762629f2424462604bd356d8e6"} Oct 30 22:45:19.369531 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.369513 4859 scope.go:122] "RemoveContainer" containerID="179c0bfdb972c1518faff7dd471ea26f70e894191c37ab8351f13dbffec5c6b2" Oct 30 22:45:19.384796 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.384777 4859 scope.go:122] "RemoveContainer" containerID="cbff1f1995640a299d6fa306181612d04b17803e51f734d64038412c66ca064f" Oct 30 22:45:19.527877 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.527838 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4"] Oct 30 22:45:19.536994 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.536961 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-z6gg4"] Oct 30 22:45:19.563510 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.563479 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl"] Oct 30 22:45:19.660223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.659012 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r"] Oct 30 22:45:19.665215 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:19.665179 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a6900c-29890005-zrl8r"] Oct 30 22:45:20.510285 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:20.510259 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" Oct 30 22:45:20.576428 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:20.576396 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-kube-api-access-2j7mn\" (UniqueName: \"kubernetes.io/projected/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-kube-api-access-2j7mn\") pod \"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3\" (UID: \"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3\") " Oct 30 22:45:20.576684 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:20.576662 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-maas-api-service-ca\") pod \"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3\" (UID: \"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3\") " Oct 30 22:45:20.577078 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:20.577034 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-maas-api-service-ca" pod "ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" (UID: "ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:20.577168 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:20.577155 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-maas-api-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:20.578586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:20.578559 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-kube-api-access-2j7mn" pod "ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" (UID: "ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3"). InnerVolumeSpecName "kube-api-access-2j7mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:20.678578 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:20.678507 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j7mn\" (UniqueName: \"kubernetes.io/projected/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3-kube-api-access-2j7mn\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:20.743341 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:20.743305 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" path="/var/lib/kubelet/pods/3aaa5067-c0fe-4992-9d93-6841065c0348/volumes" Oct 30 22:45:20.743884 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:20.743869 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" path="/var/lib/kubelet/pods/651c95b3-bdd2-4bcd-a299-73c43fe919dc/volumes" Oct 30 22:45:20.744396 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:20.744380 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" path="/var/lib/kubelet/pods/7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f/volumes" Oct 30 22:45:21.383196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:21.383165 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" Oct 30 22:45:21.383379 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:21.383214 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl" event={"ID":"ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3","Type":"ContainerDied","Data":"29c29c6a08c5f8141ea31292870f229eb411a086bc467d181d6a53c2299ef280"} Oct 30 22:45:21.383379 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:21.383279 4859 scope.go:122] "RemoveContainer" containerID="a236181199665d1736b0868c6ef51dbd3eb1eaf1fa337ae6ac57404d0a74c758" Oct 30 22:45:21.548757 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:21.548723 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl"] Oct 30 22:45:21.597419 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:21.597385 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b1eb3f-29890005-klhhl"] Oct 30 22:45:22.741167 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:22.741137 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" path="/var/lib/kubelet/pods/ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3/volumes" Oct 30 22:45:38.266316 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:38.266282 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:45:39.922147 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:45:39.922105 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:45:51.211029 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.210991 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl"] Oct 30 22:45:51.211872 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211853 4859 cpu_manager.go:414] "containerMap: removing container" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerName="cleanup" Oct 30 22:45:51.211920 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211875 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerName="cleanup" Oct 30 22:45:51.211920 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211901 4859 cpu_manager.go:414] "containerMap: removing container" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerName="cleanup" Oct 30 22:45:51.211920 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211909 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerName="cleanup" Oct 30 22:45:51.212044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211920 4859 cpu_manager.go:414] "containerMap: removing container" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerName="cleanup" Oct 30 22:45:51.212044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211928 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerName="cleanup" Oct 30 22:45:51.212044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211956 4859 cpu_manager.go:414] "containerMap: removing container" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerName="cleanup" Oct 30 22:45:51.212044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211964 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerName="cleanup" Oct 30 22:45:51.212044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211978 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerName="cleanup" Oct 30 22:45:51.212044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211985 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerName="cleanup" Oct 30 22:45:51.212044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.211998 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerName="cleanup" Oct 30 22:45:51.212044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212006 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerName="cleanup" Oct 30 22:45:51.212044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212028 4859 cpu_manager.go:414] "containerMap: removing container" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerName="cleanup" Oct 30 22:45:51.212044 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212036 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212053 4859 cpu_manager.go:414] "containerMap: removing container" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212061 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212090 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212096 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212125 4859 cpu_manager.go:414] "containerMap: removing container" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212133 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212150 4859 cpu_manager.go:414] "containerMap: removing container" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212157 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212167 4859 cpu_manager.go:414] "containerMap: removing container" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212175 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212185 4859 cpu_manager.go:414] "containerMap: removing container" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212192 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212319 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212330 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212342 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212351 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212359 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerName="cleanup" Oct 30 22:45:51.212367 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212368 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerName="cleanup" Oct 30 22:45:51.212893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212378 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerName="cleanup" Oct 30 22:45:51.212893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212391 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aaa5067-c0fe-4992-9d93-6841065c0348" containerName="cleanup" Oct 30 22:45:51.212893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212400 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerName="cleanup" Oct 30 22:45:51.212893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212409 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerName="cleanup" Oct 30 22:45:51.212893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212419 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerName="cleanup" Oct 30 22:45:51.212893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212428 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="651c95b3-bdd2-4bcd-a299-73c43fe919dc" containerName="cleanup" Oct 30 22:45:51.212893 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.212438 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerName="cleanup" Oct 30 22:45:51.217129 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.217112 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.307922 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.307888 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-9e18b5-openshift-default-dockercfg-mqdzm\"" Oct 30 22:45:51.331179 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.331138 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-podinfo\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.331356 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.331299 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvmm\" (UniqueName: \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-kube-api-access-tzvmm\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.331461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.331439 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-envoy\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.331528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.331504 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-socket\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.331587 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.331539 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-token\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.331669 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.331649 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-certs\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.331707 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.331696 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-credential-socket\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.331810 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.331796 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-data\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.331872 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.331838 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istiod-ca-cert\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.343307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.343275 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl"] Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.433344 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-data\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.433416 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istiod-ca-cert\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.433454 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-podinfo\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.433540 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvmm\" (UniqueName: \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-kube-api-access-tzvmm\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.433637 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-envoy\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.433690 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-socket\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.433719 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-token\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.433777 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-certs\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.433832 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-credential-socket\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.434400 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-credential-socket\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.434662 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-data\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.435586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.435361 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istiod-ca-cert\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.436743 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.436693 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-socket\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.436994 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.436974 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-certs\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.439244 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.439222 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-envoy\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.439575 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.439548 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-podinfo\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.493216 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.493143 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvmm\" (UniqueName: \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-kube-api-access-tzvmm\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.493216 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.493166 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-token\") pod \"e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.530194 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.530161 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:51.991703 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:45:51.991672 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc053df_ca6b_49b5_ab28_04b0fafa67e2.slice/crio-3604b425ea5cf77b62ea2530b17172501b4ec10d43d2eb5b389b9cb5a6e58de7 WatchSource:0}: Error finding container 3604b425ea5cf77b62ea2530b17172501b4ec10d43d2eb5b389b9cb5a6e58de7: Status 404 returned error can't find the container with id 3604b425ea5cf77b62ea2530b17172501b4ec10d43d2eb5b389b9cb5a6e58de7 Oct 30 22:45:51.992663 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.992635 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl"] Oct 30 22:45:51.994440 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.994405 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:45:51.994539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.994485 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:45:51.994604 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:51.994534 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:45:52.611980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:52.611924 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" event={"ID":"abc053df-ca6b-49b5-ab28-04b0fafa67e2","Type":"ContainerStarted","Data":"e11f4ee8bba0a1286147e5fdf75fa98b643def7a7ff2199b96984570a61bceb4"} Oct 30 22:45:52.612450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:52.612005 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" event={"ID":"abc053df-ca6b-49b5-ab28-04b0fafa67e2","Type":"ContainerStarted","Data":"3604b425ea5cf77b62ea2530b17172501b4ec10d43d2eb5b389b9cb5a6e58de7"} Oct 30 22:45:52.745119 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:52.745050 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" podStartSLOduration=1.7450324149999998 podStartE2EDuration="1.745032415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:52.739829054 +0000 UTC m=+3810.112455804" watchObservedRunningTime="2026-10-30 22:45:52.745032415 +0000 UTC m=+3810.117659163" Oct 30 22:45:53.530779 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:53.530743 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:53.536797 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:53.536773 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:53.618025 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:53.617988 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:45:53.619196 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:45:53.619172 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:46:09.958720 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:46:09.958628 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:46:17.374964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:17.374910 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:46:39.882655 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:46:39.882610 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:46:43.780052 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.780013 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph"] Oct 30 22:46:43.780604 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.780590 4859 cpu_manager.go:414] "containerMap: removing container" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerName="cleanup" Oct 30 22:46:43.780653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.780605 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerName="cleanup" Oct 30 22:46:43.780653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.780618 4859 cpu_manager.go:414] "containerMap: removing container" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerName="cleanup" Oct 30 22:46:43.780653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.780626 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3e3b5b-0359-4d3d-b4dc-e03597c9c89f" containerName="cleanup" Oct 30 22:46:43.780761 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.780751 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="29747294-37cb-465a-aa8f-b67d7a64c72a" containerName="cleanup" Oct 30 22:46:43.780797 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.780762 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae92f461-f0cc-4bdb-852c-b7d8d73aa4c3" containerName="cleanup" Oct 30 22:46:43.784426 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.784405 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:43.836539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.836498 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-69058c-serving-cert\"" Oct 30 22:46:43.871564 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.871516 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclnb\" (UniqueName: \"kubernetes.io/projected/082c36fb-0359-48b9-bd96-7248ee570108-kube-api-access-jclnb\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:43.871781 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.871747 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:43.871856 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.871812 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-metrics-certs\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:43.871981 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.871925 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/082c36fb-0359-48b9-bd96-7248ee570108-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:43.891315 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.891278 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph"] Oct 30 22:46:43.973543 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.973494 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclnb\" (UniqueName: \"kubernetes.io/projected/082c36fb-0359-48b9-bd96-7248ee570108-kube-api-access-jclnb\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:43.973749 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.973617 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:43.973749 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.973654 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-metrics-certs\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:43.973749 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.973687 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/082c36fb-0359-48b9-bd96-7248ee570108-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:43.973931 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:46:43.973775 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-serving-cert: secret "maas-api-e2e-extmod-b-69058c-serving-cert" not found Oct 30 22:46:43.973931 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:46:43.973874 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-maas-api-tls podName:082c36fb-0359-48b9-bd96-7248ee570108 nodeName:}" failed. No retries permitted until 2026-10-30 22:46:44.473851588 +0000 UTC m=+3861.846478328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-maas-api-tls") pod "maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" (UID: "082c36fb-0359-48b9-bd96-7248ee570108") : secret "maas-api-e2e-extmod-b-69058c-serving-cert" not found Oct 30 22:46:43.974582 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.974554 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/082c36fb-0359-48b9-bd96-7248ee570108-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:43.976171 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:43.976147 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-metrics-certs\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:44.064803 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:44.064709 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclnb\" (UniqueName: \"kubernetes.io/projected/082c36fb-0359-48b9-bd96-7248ee570108-kube-api-access-jclnb\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:44.480620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:44.480585 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:44.483130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:44.483109 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:44.698811 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:44.698770 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:45.082373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:45.082341 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph"] Oct 30 22:46:45.086017 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:46:45.085984 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082c36fb_0359_48b9_bd96_7248ee570108.slice/crio-073c338c377f4635296c0c7d3af9fe4487f366edd78e9f541685ebab596c9935 WatchSource:0}: Error finding container 073c338c377f4635296c0c7d3af9fe4487f366edd78e9f541685ebab596c9935: Status 404 returned error can't find the container with id 073c338c377f4635296c0c7d3af9fe4487f366edd78e9f541685ebab596c9935 Oct 30 22:46:45.191766 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:45.191725 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" event={"ID":"082c36fb-0359-48b9-bd96-7248ee570108","Type":"ContainerStarted","Data":"073c338c377f4635296c0c7d3af9fe4487f366edd78e9f541685ebab596c9935"} Oct 30 22:46:46.207397 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:46.207297 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" event={"ID":"082c36fb-0359-48b9-bd96-7248ee570108","Type":"ContainerStarted","Data":"410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61"} Oct 30 22:46:46.207778 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:46.207399 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:46:46.323382 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:46.323325 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" podStartSLOduration=2.46998474 podStartE2EDuration="3.3233107s" totalImagesPullingTime="853.32596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:46.321902269 +0000 UTC m=+3863.694529017" watchObservedRunningTime="2026-10-30 22:46:46.3233107 +0000 UTC m=+3863.695937447" Oct 30 22:46:52.224928 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:46:52.224896 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:47:09.880232 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:47:09.880180 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:47:10.534569 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:10.534535 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:47:22.470147 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:22.470113 4859 generic.go:356] "Generic (PLEG): container finished" podID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerID="ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad" exitCode=0 Oct 30 22:47:22.470572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:22.470179 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" event={"ID":"6d35c973-cc54-443b-b0a4-add6d36a409c","Type":"ContainerDied","Data":"ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad"} Oct 30 22:47:22.470572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:22.470550 4859 scope.go:122] "RemoveContainer" containerID="ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad" Oct 30 22:47:22.472258 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:22.472218 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:47:22.472320 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:22.472303 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:47:22.472366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:22.472336 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:47:23.451307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:23.451280 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:47:23.459266 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:23.459241 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:47:23.501908 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:23.501873 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" event={"ID":"6d35c973-cc54-443b-b0a4-add6d36a409c","Type":"ContainerStarted","Data":"1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4"} Oct 30 22:47:23.508489 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:23.508461 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:47:23.515764 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:23.515741 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:47:24.095208 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:24.095163 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:47:24.096591 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:24.096566 4859 patch_prober.go:29] interesting pod/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:47:24.096742 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:24.096624 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 22:47:25.095181 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:25.095139 4859 patch_prober.go:29] interesting pod/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:47:25.095631 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:25.095227 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 22:47:26.094963 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:26.094909 4859 patch_prober.go:29] interesting pod/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:47:26.095152 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:26.094990 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 22:47:26.233219 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:26.233182 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:47:27.095501 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:27.095463 4859 patch_prober.go:29] interesting pod/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:47:27.095702 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:27.095549 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 22:47:28.095365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:28.095324 4859 patch_prober.go:29] interesting pod/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:47:28.095874 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:28.095838 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 22:47:29.100892 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:29.100857 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:47:29.101868 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:47:29.101846 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:47:39.879123 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:47:39.879042 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:48:08.295631 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:08.295581 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:48:09.877356 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:48:09.877287 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:48:16.386758 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.386716 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d"] Oct 30 22:48:16.392187 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.392158 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.444388 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.444348 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-b1b1eb-serving-cert\"" Oct 30 22:48:16.470130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.470083 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nqd\" (UniqueName: \"kubernetes.io/projected/0ec2eb4a-892d-4311-b638-4e2330270d73-kube-api-access-86nqd\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.470338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.470140 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.470338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.470180 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-metrics-certs\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.470338 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.470306 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec2eb4a-892d-4311-b638-4e2330270d73-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.499112 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.499068 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d"] Oct 30 22:48:16.571494 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.571446 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nqd\" (UniqueName: \"kubernetes.io/projected/0ec2eb4a-892d-4311-b638-4e2330270d73-kube-api-access-86nqd\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.571494 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.571495 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.571763 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:48:16.571608 4859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-serving-cert: secret "maas-api-e2e-extmod-b-b1b1eb-serving-cert" not found Oct 30 22:48:16.571763 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:48:16.571682 4859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-maas-api-tls podName:0ec2eb4a-892d-4311-b638-4e2330270d73 nodeName:}" failed. No retries permitted until 2026-10-30 22:48:17.071663338 +0000 UTC m=+3954.444290074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-maas-api-tls") pod "maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" (UID: "0ec2eb4a-892d-4311-b638-4e2330270d73") : secret "maas-api-e2e-extmod-b-b1b1eb-serving-cert" not found Oct 30 22:48:16.571763 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.571680 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-metrics-certs\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.571980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.571899 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec2eb4a-892d-4311-b638-4e2330270d73-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.572765 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.572733 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec2eb4a-892d-4311-b638-4e2330270d73-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.574748 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.574723 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-metrics-certs\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:16.631886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:16.631820 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nqd\" (UniqueName: \"kubernetes.io/projected/0ec2eb4a-892d-4311-b638-4e2330270d73-kube-api-access-86nqd\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:17.095141 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:17.095030 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:17.103270 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:17.103231 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:17.310427 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:17.310387 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:17.705307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:17.705144 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d"] Oct 30 22:48:17.904384 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:17.904344 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" event={"ID":"0ec2eb4a-892d-4311-b638-4e2330270d73","Type":"ContainerStarted","Data":"89605ad2c19c26e9c6abd49bd1c03b0ee64dc24eac00465177924af3ef65cbdb"} Oct 30 22:48:18.915022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:18.914886 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" event={"ID":"0ec2eb4a-892d-4311-b638-4e2330270d73","Type":"ContainerStarted","Data":"e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89"} Oct 30 22:48:18.915022 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:18.914997 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:19.030425 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:19.030361 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" podStartSLOduration=2.094686318 podStartE2EDuration="3.030341768s" totalImagesPullingTime="935.65545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:19.028545314 +0000 UTC m=+3956.401172057" watchObservedRunningTime="2026-10-30 22:48:19.030341768 +0000 UTC m=+3956.402968515" Oct 30 22:48:19.924792 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:19.924760 4859 generic.go:356] "Generic (PLEG): container finished" podID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerID="e11f4ee8bba0a1286147e5fdf75fa98b643def7a7ff2199b96984570a61bceb4" exitCode=0 Oct 30 22:48:19.925242 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:19.924841 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" event={"ID":"abc053df-ca6b-49b5-ab28-04b0fafa67e2","Type":"ContainerDied","Data":"e11f4ee8bba0a1286147e5fdf75fa98b643def7a7ff2199b96984570a61bceb4"} Oct 30 22:48:19.925378 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:19.925359 4859 scope.go:122] "RemoveContainer" containerID="e11f4ee8bba0a1286147e5fdf75fa98b643def7a7ff2199b96984570a61bceb4" Oct 30 22:48:19.927086 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:19.927060 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:48:19.927222 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:19.927126 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:48:19.927222 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:19.927161 4859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:48:20.934265 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:20.934229 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" event={"ID":"abc053df-ca6b-49b5-ab28-04b0fafa67e2","Type":"ContainerStarted","Data":"ddbe03382669a8044cca2925795fef8a7f4d792b3474ba0ef11a7a4c7d82a786"} Oct 30 22:48:21.530516 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:21.530482 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:48:21.532140 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:21.532101 4859 patch_prober.go:29] interesting pod/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:48:21.532315 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:21.532162 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 22:48:22.530864 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:22.530827 4859 patch_prober.go:29] interesting pod/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:48:22.531353 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:22.530881 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 22:48:23.530537 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:23.530501 4859 patch_prober.go:29] interesting pod/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:48:23.530719 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:23.530551 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 22:48:23.618962 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:23.618913 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:48:24.530461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:24.530429 4859 patch_prober.go:29] interesting pod/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:48:24.530675 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:24.530482 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 22:48:24.929682 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:24.929652 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:48:25.531207 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:25.531171 4859 patch_prober.go:29] interesting pod/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:48:25.531404 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:25.531228 4859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 22:48:26.535835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:26.535806 4859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:48:26.536725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:26.536702 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:48:39.877583 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:48:39.877549 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:48:42.874357 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:48:42.874311 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:49:09.879078 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:49:09.878971 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:49:39.544451 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:39.544417 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:49:39.877668 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:49:39.877626 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:49:45.649465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:45.649422 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9"] Oct 30 22:49:45.650158 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:45.649709 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" podUID="336cf7be-cb8c-4938-b19e-e37f6f99b165" containerName="maas-api" containerID="cri-o://3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab" gracePeriod=30 Oct 30 22:49:46.002753 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.002725 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:49:46.099725 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.099696 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336cf7be-cb8c-4938-b19e-e37f6f99b165-kube-api-access-77wkf\" (UniqueName: \"kubernetes.io/projected/336cf7be-cb8c-4938-b19e-e37f6f99b165-kube-api-access-77wkf\") pod \"336cf7be-cb8c-4938-b19e-e37f6f99b165\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " Oct 30 22:49:46.099919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.099897 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-metrics-certs\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-metrics-certs\") pod \"336cf7be-cb8c-4938-b19e-e37f6f99b165\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " Oct 30 22:49:46.100058 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.100029 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-maas-api-tls\") pod \"336cf7be-cb8c-4938-b19e-e37f6f99b165\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " Oct 30 22:49:46.100199 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.100104 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336cf7be-cb8c-4938-b19e-e37f6f99b165-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336cf7be-cb8c-4938-b19e-e37f6f99b165-openshift-service-ca\") pod \"336cf7be-cb8c-4938-b19e-e37f6f99b165\" (UID: \"336cf7be-cb8c-4938-b19e-e37f6f99b165\") " Oct 30 22:49:46.100465 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.100435 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336cf7be-cb8c-4938-b19e-e37f6f99b165-openshift-service-ca" pod "336cf7be-cb8c-4938-b19e-e37f6f99b165" (UID: "336cf7be-cb8c-4938-b19e-e37f6f99b165"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:46.100629 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.100612 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336cf7be-cb8c-4938-b19e-e37f6f99b165-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:46.102356 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.102331 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-metrics-certs" pod "336cf7be-cb8c-4938-b19e-e37f6f99b165" (UID: "336cf7be-cb8c-4938-b19e-e37f6f99b165"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:46.102447 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.102360 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336cf7be-cb8c-4938-b19e-e37f6f99b165-kube-api-access-77wkf" pod "336cf7be-cb8c-4938-b19e-e37f6f99b165" (UID: "336cf7be-cb8c-4938-b19e-e37f6f99b165"). InnerVolumeSpecName "kube-api-access-77wkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:46.102572 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.102545 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-maas-api-tls" pod "336cf7be-cb8c-4938-b19e-e37f6f99b165" (UID: "336cf7be-cb8c-4938-b19e-e37f6f99b165"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:46.201743 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.201707 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77wkf\" (UniqueName: \"kubernetes.io/projected/336cf7be-cb8c-4938-b19e-e37f6f99b165-kube-api-access-77wkf\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:46.201743 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.201738 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:46.201743 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.201748 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336cf7be-cb8c-4938-b19e-e37f6f99b165-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:46.694476 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.694429 4859 generic.go:356] "Generic (PLEG): container finished" podID="336cf7be-cb8c-4938-b19e-e37f6f99b165" containerID="3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab" exitCode=0 Oct 30 22:49:46.694990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.694642 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" event={"ID":"336cf7be-cb8c-4938-b19e-e37f6f99b165","Type":"ContainerDied","Data":"3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab"} Oct 30 22:49:46.694990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.694692 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" event={"ID":"336cf7be-cb8c-4938-b19e-e37f6f99b165","Type":"ContainerDied","Data":"a91ba391a03515d538231364a1a8fa147eb4d6ad2ea015c4647ad557df30113b"} Oct 30 22:49:46.694990 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.694719 4859 scope.go:122] "RemoveContainer" containerID="3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab" Oct 30 22:49:46.696248 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.696002 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9" Oct 30 22:49:46.708184 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.708162 4859 scope.go:122] "RemoveContainer" containerID="3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab" Oct 30 22:49:46.708519 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:49:46.708488 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab\": container with ID starting with 3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab not found: ID does not exist" containerID="3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab" Oct 30 22:49:46.708648 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.708529 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab"} err="failed to get container status \"3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab\": rpc error: code = NotFound desc = could not find container \"3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab\": container with ID starting with 3e1378dafe0058ccf02345c5dface3595b652b2f037481e851dcbc6973f7efab not found: ID does not exist" Oct 30 22:49:46.861563 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.861533 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9"] Oct 30 22:49:46.910550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:46.910520 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-15e0f6-cb88c8b4d-gc9n9"] Oct 30 22:49:48.743829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:48.743787 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336cf7be-cb8c-4938-b19e-e37f6f99b165" path="/var/lib/kubelet/pods/336cf7be-cb8c-4938-b19e-e37f6f99b165/volumes" Oct 30 22:49:53.507816 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:53.507778 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl"] Oct 30 22:49:53.508308 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:53.508018 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" containerID="cri-o://ddbe03382669a8044cca2925795fef8a7f4d792b3474ba0ef11a7a4c7d82a786" gracePeriod=30 Oct 30 22:49:56.082826 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:56.082600 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8"] Oct 30 22:49:56.083348 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:56.082864 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" containerID="cri-o://bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57" gracePeriod=30 Oct 30 22:49:56.538484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:56.538448 4859 patch_prober.go:29] interesting pod/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:49:56.538697 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:56.538525 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:49:58.788542 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:58.788464 4859 generic.go:356] "Generic (PLEG): container finished" podID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerID="ddbe03382669a8044cca2925795fef8a7f4d792b3474ba0ef11a7a4c7d82a786" exitCode=0 Oct 30 22:49:58.788988 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:58.788548 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" event={"ID":"abc053df-ca6b-49b5-ab28-04b0fafa67e2","Type":"ContainerDied","Data":"ddbe03382669a8044cca2925795fef8a7f4d792b3474ba0ef11a7a4c7d82a786"} Oct 30 22:49:58.788988 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:58.788607 4859 scope.go:122] "RemoveContainer" containerID="e11f4ee8bba0a1286147e5fdf75fa98b643def7a7ff2199b96984570a61bceb4" Oct 30 22:49:58.885860 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:58.885839 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:49:59.031024 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.030986 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-envoy\") pod \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " Oct 30 22:49:59.031187 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031035 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-certs\") pod \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " Oct 30 22:49:59.031187 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031113 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-data\") pod \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " Oct 30 22:49:59.031187 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031141 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-podinfo\") pod \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " Oct 30 22:49:59.031366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031223 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-kube-api-access-tzvmm\" (UniqueName: \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-kube-api-access-tzvmm\") pod \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " Oct 30 22:49:59.031366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031250 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-credential-socket\") pod \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " Oct 30 22:49:59.031366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031321 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istiod-ca-cert\") pod \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " Oct 30 22:49:59.031366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031327 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-certs" pod "abc053df-ca6b-49b5-ab28-04b0fafa67e2" (UID: "abc053df-ca6b-49b5-ab28-04b0fafa67e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:59.031550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031369 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-token\" (UniqueName: \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-token\") pod \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " Oct 30 22:49:59.031550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031403 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-socket\") pod \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\" (UID: \"abc053df-ca6b-49b5-ab28-04b0fafa67e2\") " Oct 30 22:49:59.031550 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031447 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-data" pod "abc053df-ca6b-49b5-ab28-04b0fafa67e2" (UID: "abc053df-ca6b-49b5-ab28-04b0fafa67e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:59.031700 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031547 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-credential-socket" pod "abc053df-ca6b-49b5-ab28-04b0fafa67e2" (UID: "abc053df-ca6b-49b5-ab28-04b0fafa67e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:59.031783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031755 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istiod-ca-cert" pod "abc053df-ca6b-49b5-ab28-04b0fafa67e2" (UID: "abc053df-ca6b-49b5-ab28-04b0fafa67e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:59.032003 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.031980 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-socket" pod "abc053df-ca6b-49b5-ab28-04b0fafa67e2" (UID: "abc053df-ca6b-49b5-ab28-04b0fafa67e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:59.032117 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.032076 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:59.032117 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.032096 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:59.032117 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.032109 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:59.032296 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.032123 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:59.033473 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.033446 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-envoy" pod "abc053df-ca6b-49b5-ab28-04b0fafa67e2" (UID: "abc053df-ca6b-49b5-ab28-04b0fafa67e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:59.033691 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.033664 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-podinfo" pod "abc053df-ca6b-49b5-ab28-04b0fafa67e2" (UID: "abc053df-ca6b-49b5-ab28-04b0fafa67e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:49:59.033960 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.033924 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-token" pod "abc053df-ca6b-49b5-ab28-04b0fafa67e2" (UID: "abc053df-ca6b-49b5-ab28-04b0fafa67e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:59.034458 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.034438 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-kube-api-access-tzvmm" pod "abc053df-ca6b-49b5-ab28-04b0fafa67e2" (UID: "abc053df-ca6b-49b5-ab28-04b0fafa67e2"). InnerVolumeSpecName "kube-api-access-tzvmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:59.133756 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.133727 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:59.133756 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.133754 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:59.133920 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.133764 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzvmm\" (UniqueName: \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-kube-api-access-tzvmm\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:59.133920 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.133773 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc053df-ca6b-49b5-ab28-04b0fafa67e2-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:59.133920 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.133782 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc053df-ca6b-49b5-ab28-04b0fafa67e2-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:59.798110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.798077 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" event={"ID":"abc053df-ca6b-49b5-ab28-04b0fafa67e2","Type":"ContainerDied","Data":"3604b425ea5cf77b62ea2530b17172501b4ec10d43d2eb5b389b9cb5a6e58de7"} Oct 30 22:49:59.798543 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.798133 4859 scope.go:122] "RemoveContainer" containerID="ddbe03382669a8044cca2925795fef8a7f4d792b3474ba0ef11a7a4c7d82a786" Oct 30 22:49:59.798543 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:49:59.798137 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl" Oct 30 22:50:00.019310 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:00.019276 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl"] Oct 30 22:50:00.070126 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:00.070054 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-9e18b5-openshift-default-54dd5bf5cf-f84tl"] Oct 30 22:50:00.308829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:00.289332 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:50:00.308829 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:00.289392 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:50:00.741761 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:00.741728 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" path="/var/lib/kubelet/pods/abc053df-ca6b-49b5-ab28-04b0fafa67e2/volumes" Oct 30 22:50:01.353364 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.353306 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:50:01.462890 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.462858 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-podinfo\") pod \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " Oct 30 22:50:01.463110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.462929 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-kube-api-access-w5df7\" (UniqueName: \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-kube-api-access-w5df7\") pod \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " Oct 30 22:50:01.463110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.462970 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-socket\") pod \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " Oct 30 22:50:01.463110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463014 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istiod-ca-cert\") pod \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " Oct 30 22:50:01.463110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463048 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-certs\") pod \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " Oct 30 22:50:01.463110 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463108 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-envoy\") pod \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " Oct 30 22:50:01.463378 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463222 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-token\" (UniqueName: \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-token\") pod \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " Oct 30 22:50:01.463378 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463319 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-socket" pod "825d33d5-6db2-4c6b-a8a9-bd5a8a615449" (UID: "825d33d5-6db2-4c6b-a8a9-bd5a8a615449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:01.463378 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463336 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-credential-socket\") pod \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " Oct 30 22:50:01.463535 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463375 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-data\") pod \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\" (UID: \"825d33d5-6db2-4c6b-a8a9-bd5a8a615449\") " Oct 30 22:50:01.463535 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463475 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istiod-ca-cert" pod "825d33d5-6db2-4c6b-a8a9-bd5a8a615449" (UID: "825d33d5-6db2-4c6b-a8a9-bd5a8a615449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:50:01.463894 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463760 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-certs" pod "825d33d5-6db2-4c6b-a8a9-bd5a8a615449" (UID: "825d33d5-6db2-4c6b-a8a9-bd5a8a615449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:01.463894 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463862 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:01.463894 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463867 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-credential-socket" pod "825d33d5-6db2-4c6b-a8a9-bd5a8a615449" (UID: "825d33d5-6db2-4c6b-a8a9-bd5a8a615449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:01.463894 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463882 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:01.463894 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.463896 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:01.464227 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.464024 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-data" pod "825d33d5-6db2-4c6b-a8a9-bd5a8a615449" (UID: "825d33d5-6db2-4c6b-a8a9-bd5a8a615449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:01.465655 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.465632 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-envoy" pod "825d33d5-6db2-4c6b-a8a9-bd5a8a615449" (UID: "825d33d5-6db2-4c6b-a8a9-bd5a8a615449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:01.465762 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.465631 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-podinfo" pod "825d33d5-6db2-4c6b-a8a9-bd5a8a615449" (UID: "825d33d5-6db2-4c6b-a8a9-bd5a8a615449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:50:01.466032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.466011 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-kube-api-access-w5df7" pod "825d33d5-6db2-4c6b-a8a9-bd5a8a615449" (UID: "825d33d5-6db2-4c6b-a8a9-bd5a8a615449"). InnerVolumeSpecName "kube-api-access-w5df7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:01.466972 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.466931 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-token" pod "825d33d5-6db2-4c6b-a8a9-bd5a8a615449" (UID: "825d33d5-6db2-4c6b-a8a9-bd5a8a615449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:01.565514 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.565474 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:01.565514 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.565504 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5df7\" (UniqueName: \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-kube-api-access-w5df7\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:01.565514 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.565516 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:01.565514 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.565525 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:01.565870 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.565537 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:01.565870 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.565544 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825d33d5-6db2-4c6b-a8a9-bd5a8a615449-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:01.815887 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.815798 4859 generic.go:356] "Generic (PLEG): container finished" podID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerID="bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57" exitCode=0 Oct 30 22:50:01.815887 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.815862 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" Oct 30 22:50:01.816087 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.815879 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" event={"ID":"825d33d5-6db2-4c6b-a8a9-bd5a8a615449","Type":"ContainerDied","Data":"bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57"} Oct 30 22:50:01.816087 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.815939 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8" event={"ID":"825d33d5-6db2-4c6b-a8a9-bd5a8a615449","Type":"ContainerDied","Data":"cd876bd590f0364fe32164e9e6331bdce74af2218283f7dc1ce9f6c09252f1a3"} Oct 30 22:50:01.816087 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.815967 4859 scope.go:122] "RemoveContainer" containerID="bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57" Oct 30 22:50:01.831745 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.831720 4859 scope.go:122] "RemoveContainer" containerID="5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a" Oct 30 22:50:01.841584 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.841566 4859 scope.go:122] "RemoveContainer" containerID="bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57" Oct 30 22:50:01.841870 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:50:01.841854 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57\": container with ID starting with bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57 not found: ID does not exist" containerID="bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57" Oct 30 22:50:01.841969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.841880 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57"} err="failed to get container status \"bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57\": rpc error: code = NotFound desc = could not find container \"bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57\": container with ID starting with bc4888edeef21712788bf52a257d96991368a9c47172b510889110dffe3a3f57 not found: ID does not exist" Oct 30 22:50:01.841969 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.841896 4859 scope.go:122] "RemoveContainer" containerID="5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a" Oct 30 22:50:01.842192 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:50:01.842172 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a\": container with ID starting with 5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a not found: ID does not exist" containerID="5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a" Oct 30 22:50:01.842234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.842200 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a"} err="failed to get container status \"5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a\": rpc error: code = NotFound desc = could not find container \"5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a\": container with ID starting with 5b53c6632c999174aeb55768ce03070aa8ffbd2e9ab114c6818ec4ee8ce39b1a not found: ID does not exist" Oct 30 22:50:01.980883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:01.980847 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8"] Oct 30 22:50:02.029715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:02.029681 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a6900c-openshift-default-59799ddcb9-2hcb8"] Oct 30 22:50:02.742278 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:02.742245 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" path="/var/lib/kubelet/pods/825d33d5-6db2-4c6b-a8a9-bd5a8a615449/volumes" Oct 30 22:50:09.877603 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:50:09.877566 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:50:13.380545 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:13.380504 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf"] Oct 30 22:50:13.381459 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:13.381422 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" containerID="cri-o://20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a" gracePeriod=30 Oct 30 22:50:15.232608 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:15.232573 4859 patch_prober.go:29] interesting pod/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:50:15.233015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:15.232627 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:50:18.656820 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.656759 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:50:18.703413 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703374 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-socket\") pod \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " Oct 30 22:50:18.703577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703446 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-certs\") pod \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " Oct 30 22:50:18.703577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703472 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-data\") pod \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " Oct 30 22:50:18.703577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703497 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-kube-api-access-cfk7z\" (UniqueName: \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-kube-api-access-cfk7z\") pod \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " Oct 30 22:50:18.703577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703515 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-credential-socket\") pod \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " Oct 30 22:50:18.703577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703575 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istiod-ca-cert\") pod \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " Oct 30 22:50:18.703856 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703650 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-podinfo\") pod \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " Oct 30 22:50:18.703856 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703691 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-socket" pod "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" (UID: "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:18.703856 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703777 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-token\" (UniqueName: \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-token\") pod \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " Oct 30 22:50:18.703856 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703817 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-envoy\") pod \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\" (UID: \"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b\") " Oct 30 22:50:18.704183 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703817 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-certs" pod "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" (UID: "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:18.704385 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.703854 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-data" pod "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" (UID: "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:18.704499 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.704181 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istiod-ca-cert" pod "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" (UID: "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:50:18.704608 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.704409 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-credential-socket" pod "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" (UID: "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:18.704608 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.704412 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:18.706159 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.706135 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-podinfo" pod "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" (UID: "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:50:18.706247 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.706192 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-kube-api-access-cfk7z" pod "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" (UID: "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b"). InnerVolumeSpecName "kube-api-access-cfk7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:18.706382 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.706367 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-envoy" pod "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" (UID: "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:18.707252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.707228 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-token" pod "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" (UID: "e3f42837-1b1c-4f5b-9236-16ed9c6cb95b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:18.806321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.806289 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:18.806321 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.806323 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:18.806539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.806339 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:18.806539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.806353 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:18.806539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.806366 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfk7z\" (UniqueName: \"kubernetes.io/projected/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-kube-api-access-cfk7z\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:18.806539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.806381 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:18.806539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.806394 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:18.806539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.806409 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:18.964503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.964435 4859 generic.go:356] "Generic (PLEG): container finished" podID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerID="20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a" exitCode=0 Oct 30 22:50:18.964503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.964489 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" Oct 30 22:50:18.964715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.964524 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" event={"ID":"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b","Type":"ContainerDied","Data":"20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a"} Oct 30 22:50:18.964715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.964587 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf" event={"ID":"e3f42837-1b1c-4f5b-9236-16ed9c6cb95b","Type":"ContainerDied","Data":"75e6c3adf80e337b9880b58ab7bea44c855c67e8f736a271ae7346e10b95dea5"} Oct 30 22:50:18.964715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.964617 4859 scope.go:122] "RemoveContainer" containerID="20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a" Oct 30 22:50:18.974678 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.974661 4859 scope.go:122] "RemoveContainer" containerID="b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4" Oct 30 22:50:18.984099 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.984084 4859 scope.go:122] "RemoveContainer" containerID="20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a" Oct 30 22:50:18.984347 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:50:18.984329 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a\": container with ID starting with 20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a not found: ID does not exist" containerID="20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a" Oct 30 22:50:18.984384 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.984359 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a"} err="failed to get container status \"20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a\": rpc error: code = NotFound desc = could not find container \"20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a\": container with ID starting with 20917eccea2ddac6f3867470f1cf168e478053f46a1d65187f5256958758de6a not found: ID does not exist" Oct 30 22:50:18.984384 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.984378 4859 scope.go:122] "RemoveContainer" containerID="b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4" Oct 30 22:50:18.984619 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:50:18.984602 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4\": container with ID starting with b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4 not found: ID does not exist" containerID="b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4" Oct 30 22:50:18.984665 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:18.984626 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4"} err="failed to get container status \"b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4\": rpc error: code = NotFound desc = could not find container \"b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4\": container with ID starting with b13bab9616743e0c71c9018307486f000d30ee86d7c288e215670ebbf5cf58a4 not found: ID does not exist" Oct 30 22:50:19.129453 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:19.129427 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf"] Oct 30 22:50:19.132218 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:19.132200 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-20ccef-openshift-default-67bf8f7b9-lb5sf"] Oct 30 22:50:20.742144 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:20.742110 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" path="/var/lib/kubelet/pods/e3f42837-1b1c-4f5b-9236-16ed9c6cb95b/volumes" Oct 30 22:50:35.517693 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:50:35.517618 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:50:39.876402 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:50:39.876366 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:51:07.545115 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:07.545081 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:51:08.176677 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.176640 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk"] Oct 30 22:51:08.176894 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.176857 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" podUID="e68cb285-6e5e-4918-9962-fd8cc8e8680f" containerName="maas-api" containerID="cri-o://39fdd0e75b9a2861e1a196a02e44ee683ba3237627172740de845fc9cfc69bc3" gracePeriod=30 Oct 30 22:51:08.340683 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.340647 4859 generic.go:356] "Generic (PLEG): container finished" podID="e68cb285-6e5e-4918-9962-fd8cc8e8680f" containerID="39fdd0e75b9a2861e1a196a02e44ee683ba3237627172740de845fc9cfc69bc3" exitCode=0 Oct 30 22:51:08.340869 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.340720 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" event={"ID":"e68cb285-6e5e-4918-9962-fd8cc8e8680f","Type":"ContainerDied","Data":"39fdd0e75b9a2861e1a196a02e44ee683ba3237627172740de845fc9cfc69bc3"} Oct 30 22:51:08.441408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.441342 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:51:08.555364 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.555325 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68cb285-6e5e-4918-9962-fd8cc8e8680f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68cb285-6e5e-4918-9962-fd8cc8e8680f-openshift-service-ca\") pod \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " Oct 30 22:51:08.555364 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.555368 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68cb285-6e5e-4918-9962-fd8cc8e8680f-kube-api-access-z2hhn\" (UniqueName: \"kubernetes.io/projected/e68cb285-6e5e-4918-9962-fd8cc8e8680f-kube-api-access-z2hhn\") pod \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " Oct 30 22:51:08.555835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.555420 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-metrics-certs\") pod \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " Oct 30 22:51:08.555835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.555449 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-maas-api-tls\") pod \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\" (UID: \"e68cb285-6e5e-4918-9962-fd8cc8e8680f\") " Oct 30 22:51:08.555835 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.555767 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68cb285-6e5e-4918-9962-fd8cc8e8680f-openshift-service-ca" pod "e68cb285-6e5e-4918-9962-fd8cc8e8680f" (UID: "e68cb285-6e5e-4918-9962-fd8cc8e8680f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:08.556117 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.556098 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68cb285-6e5e-4918-9962-fd8cc8e8680f-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:08.557783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.557761 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-maas-api-tls" pod "e68cb285-6e5e-4918-9962-fd8cc8e8680f" (UID: "e68cb285-6e5e-4918-9962-fd8cc8e8680f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:08.558082 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.558060 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-metrics-certs" pod "e68cb285-6e5e-4918-9962-fd8cc8e8680f" (UID: "e68cb285-6e5e-4918-9962-fd8cc8e8680f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:08.558325 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.558300 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68cb285-6e5e-4918-9962-fd8cc8e8680f-kube-api-access-z2hhn" pod "e68cb285-6e5e-4918-9962-fd8cc8e8680f" (UID: "e68cb285-6e5e-4918-9962-fd8cc8e8680f"). InnerVolumeSpecName "kube-api-access-z2hhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:08.657381 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.657347 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2hhn\" (UniqueName: \"kubernetes.io/projected/e68cb285-6e5e-4918-9962-fd8cc8e8680f-kube-api-access-z2hhn\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:08.657381 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.657376 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:08.657381 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:08.657388 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68cb285-6e5e-4918-9962-fd8cc8e8680f-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:09.349223 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.349189 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" event={"ID":"e68cb285-6e5e-4918-9962-fd8cc8e8680f","Type":"ContainerDied","Data":"c7be44911037d6b74d73640267fd1b2274ac483c8d04a86516ae541b8ea099d6"} Oct 30 22:51:09.349430 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.349256 4859 scope.go:122] "RemoveContainer" containerID="39fdd0e75b9a2861e1a196a02e44ee683ba3237627172740de845fc9cfc69bc3" Oct 30 22:51:09.349430 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.349302 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk" Oct 30 22:51:09.363499 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.363477 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d"] Oct 30 22:51:09.363709 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.363687 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" podUID="0ec2eb4a-892d-4311-b638-4e2330270d73" containerName="maas-api" containerID="cri-o://e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89" gracePeriod=30 Oct 30 22:51:09.650304 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.650281 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:51:09.656615 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.656590 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk"] Oct 30 22:51:09.659692 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.659668 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9f7654-7465777bc8-kvpmk"] Oct 30 22:51:09.686804 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.686783 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-maas-api-tls\") pod \"0ec2eb4a-892d-4311-b638-4e2330270d73\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " Oct 30 22:51:09.686925 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.686860 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec2eb4a-892d-4311-b638-4e2330270d73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec2eb4a-892d-4311-b638-4e2330270d73-openshift-service-ca\") pod \"0ec2eb4a-892d-4311-b638-4e2330270d73\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " Oct 30 22:51:09.687023 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.686983 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec2eb4a-892d-4311-b638-4e2330270d73-kube-api-access-86nqd\" (UniqueName: \"kubernetes.io/projected/0ec2eb4a-892d-4311-b638-4e2330270d73-kube-api-access-86nqd\") pod \"0ec2eb4a-892d-4311-b638-4e2330270d73\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " Oct 30 22:51:09.687142 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.687116 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-metrics-certs\") pod \"0ec2eb4a-892d-4311-b638-4e2330270d73\" (UID: \"0ec2eb4a-892d-4311-b638-4e2330270d73\") " Oct 30 22:51:09.687259 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.687204 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec2eb4a-892d-4311-b638-4e2330270d73-openshift-service-ca" pod "0ec2eb4a-892d-4311-b638-4e2330270d73" (UID: "0ec2eb4a-892d-4311-b638-4e2330270d73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:09.687480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.687465 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec2eb4a-892d-4311-b638-4e2330270d73-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:09.688910 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.688884 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-maas-api-tls" pod "0ec2eb4a-892d-4311-b638-4e2330270d73" (UID: "0ec2eb4a-892d-4311-b638-4e2330270d73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:09.689057 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.688967 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec2eb4a-892d-4311-b638-4e2330270d73-kube-api-access-86nqd" pod "0ec2eb4a-892d-4311-b638-4e2330270d73" (UID: "0ec2eb4a-892d-4311-b638-4e2330270d73"). InnerVolumeSpecName "kube-api-access-86nqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:09.689057 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.689013 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-metrics-certs" pod "0ec2eb4a-892d-4311-b638-4e2330270d73" (UID: "0ec2eb4a-892d-4311-b638-4e2330270d73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:09.788986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.788931 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:09.789141 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.788991 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec2eb4a-892d-4311-b638-4e2330270d73-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:09.789141 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:09.789008 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86nqd\" (UniqueName: \"kubernetes.io/projected/0ec2eb4a-892d-4311-b638-4e2330270d73-kube-api-access-86nqd\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:09.876821 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:51:09.876733 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:51:10.358162 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.358122 4859 generic.go:356] "Generic (PLEG): container finished" podID="0ec2eb4a-892d-4311-b638-4e2330270d73" containerID="e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89" exitCode=0 Oct 30 22:51:10.358373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.358180 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" Oct 30 22:51:10.358373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.358300 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" event={"ID":"0ec2eb4a-892d-4311-b638-4e2330270d73","Type":"ContainerDied","Data":"e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89"} Oct 30 22:51:10.358373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.358339 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d" event={"ID":"0ec2eb4a-892d-4311-b638-4e2330270d73","Type":"ContainerDied","Data":"89605ad2c19c26e9c6abd49bd1c03b0ee64dc24eac00465177924af3ef65cbdb"} Oct 30 22:51:10.358373 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.358360 4859 scope.go:122] "RemoveContainer" containerID="e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89" Oct 30 22:51:10.369830 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.369811 4859 scope.go:122] "RemoveContainer" containerID="e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89" Oct 30 22:51:10.370154 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:51:10.370103 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89\": container with ID starting with e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89 not found: ID does not exist" containerID="e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89" Oct 30 22:51:10.370246 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.370164 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89"} err="failed to get container status \"e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89\": rpc error: code = NotFound desc = could not find container \"e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89\": container with ID starting with e4576049fc0f62275cd40b2a912d9235d73eb13f923324e9cc0cc2fa5195fa89 not found: ID does not exist" Oct 30 22:51:10.525900 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.525869 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d"] Oct 30 22:51:10.575408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.575383 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b1b1eb-6f4b56ffff-9nc2d"] Oct 30 22:51:10.741466 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.741399 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec2eb4a-892d-4311-b638-4e2330270d73" path="/var/lib/kubelet/pods/0ec2eb4a-892d-4311-b638-4e2330270d73/volumes" Oct 30 22:51:10.741820 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:10.741806 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68cb285-6e5e-4918-9962-fd8cc8e8680f" path="/var/lib/kubelet/pods/e68cb285-6e5e-4918-9962-fd8cc8e8680f/volumes" Oct 30 22:51:15.597521 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.597476 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph"] Oct 30 22:51:15.598076 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.597708 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" podUID="082c36fb-0359-48b9-bd96-7248ee570108" containerName="maas-api" containerID="cri-o://410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61" gracePeriod=30 Oct 30 22:51:15.900085 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.900054 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:51:15.974291 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.974257 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/082c36fb-0359-48b9-bd96-7248ee570108-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/082c36fb-0359-48b9-bd96-7248ee570108-openshift-service-ca\") pod \"082c36fb-0359-48b9-bd96-7248ee570108\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " Oct 30 22:51:15.974452 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.974366 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082c36fb-0359-48b9-bd96-7248ee570108-kube-api-access-jclnb\" (UniqueName: \"kubernetes.io/projected/082c36fb-0359-48b9-bd96-7248ee570108-kube-api-access-jclnb\") pod \"082c36fb-0359-48b9-bd96-7248ee570108\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " Oct 30 22:51:15.974452 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.974447 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-maas-api-tls\") pod \"082c36fb-0359-48b9-bd96-7248ee570108\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " Oct 30 22:51:15.974582 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.974477 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-metrics-certs\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-metrics-certs\") pod \"082c36fb-0359-48b9-bd96-7248ee570108\" (UID: \"082c36fb-0359-48b9-bd96-7248ee570108\") " Oct 30 22:51:15.974680 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.974653 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082c36fb-0359-48b9-bd96-7248ee570108-openshift-service-ca" pod "082c36fb-0359-48b9-bd96-7248ee570108" (UID: "082c36fb-0359-48b9-bd96-7248ee570108"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:15.974934 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.974912 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/082c36fb-0359-48b9-bd96-7248ee570108-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:15.976488 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.976460 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-maas-api-tls" pod "082c36fb-0359-48b9-bd96-7248ee570108" (UID: "082c36fb-0359-48b9-bd96-7248ee570108"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:15.976590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.976474 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082c36fb-0359-48b9-bd96-7248ee570108-kube-api-access-jclnb" pod "082c36fb-0359-48b9-bd96-7248ee570108" (UID: "082c36fb-0359-48b9-bd96-7248ee570108"). InnerVolumeSpecName "kube-api-access-jclnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:15.976590 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:15.976503 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-metrics-certs" pod "082c36fb-0359-48b9-bd96-7248ee570108" (UID: "082c36fb-0359-48b9-bd96-7248ee570108"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:16.099687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.099638 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jclnb\" (UniqueName: \"kubernetes.io/projected/082c36fb-0359-48b9-bd96-7248ee570108-kube-api-access-jclnb\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:16.099687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.099676 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:16.099687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.099690 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/082c36fb-0359-48b9-bd96-7248ee570108-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:16.403007 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.402973 4859 generic.go:356] "Generic (PLEG): container finished" podID="082c36fb-0359-48b9-bd96-7248ee570108" containerID="410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61" exitCode=0 Oct 30 22:51:16.403215 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.403013 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" Oct 30 22:51:16.403215 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.403056 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" event={"ID":"082c36fb-0359-48b9-bd96-7248ee570108","Type":"ContainerDied","Data":"410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61"} Oct 30 22:51:16.403215 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.403113 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph" event={"ID":"082c36fb-0359-48b9-bd96-7248ee570108","Type":"ContainerDied","Data":"073c338c377f4635296c0c7d3af9fe4487f366edd78e9f541685ebab596c9935"} Oct 30 22:51:16.403215 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.403144 4859 scope.go:122] "RemoveContainer" containerID="410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61" Oct 30 22:51:16.413685 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.413669 4859 scope.go:122] "RemoveContainer" containerID="410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61" Oct 30 22:51:16.414016 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:51:16.413999 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61\": container with ID starting with 410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61 not found: ID does not exist" containerID="410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61" Oct 30 22:51:16.414090 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.414025 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61"} err="failed to get container status \"410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61\": rpc error: code = NotFound desc = could not find container \"410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61\": container with ID starting with 410ed0447c9f4f63268557b18c11bf92a39ef55bb8151520c866e6933e761d61 not found: ID does not exist" Oct 30 22:51:16.566261 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.566238 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph"] Oct 30 22:51:16.569279 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.569256 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69058c-5b689c8dc-n9nph"] Oct 30 22:51:16.741723 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:16.741638 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="082c36fb-0359-48b9-bd96-7248ee570108" path="/var/lib/kubelet/pods/082c36fb-0359-48b9-bd96-7248ee570108/volumes" Oct 30 22:51:24.765149 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:24.765114 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck"] Oct 30 22:51:24.765544 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:24.765305 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" containerID="cri-o://1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4" gracePeriod=30 Oct 30 22:51:29.103837 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:29.103797 4859 patch_prober.go:29] interesting pod/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:51:29.104349 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:29.103860 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:51:30.041698 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.041644 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:51:30.098718 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.098692 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-credential-socket\") pod \"6d35c973-cc54-443b-b0a4-add6d36a409c\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " Oct 30 22:51:30.098843 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.098737 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-data\") pod \"6d35c973-cc54-443b-b0a4-add6d36a409c\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " Oct 30 22:51:30.098843 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.098779 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d35c973-cc54-443b-b0a4-add6d36a409c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d35c973-cc54-443b-b0a4-add6d36a409c-istiod-ca-cert\") pod \"6d35c973-cc54-443b-b0a4-add6d36a409c\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " Oct 30 22:51:30.098843 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.098808 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-kube-api-access-86hcd\" (UniqueName: \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-kube-api-access-86hcd\") pod \"6d35c973-cc54-443b-b0a4-add6d36a409c\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " Oct 30 22:51:30.099092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.098846 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-certs\") pod \"6d35c973-cc54-443b-b0a4-add6d36a409c\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " Oct 30 22:51:30.099092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.098896 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-socket\") pod \"6d35c973-cc54-443b-b0a4-add6d36a409c\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " Oct 30 22:51:30.099092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.098926 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-envoy\") pod \"6d35c973-cc54-443b-b0a4-add6d36a409c\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " Oct 30 22:51:30.099092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.098977 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-token\" (UniqueName: \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-token\") pod \"6d35c973-cc54-443b-b0a4-add6d36a409c\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " Oct 30 22:51:30.099092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.098979 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-credential-socket" pod "6d35c973-cc54-443b-b0a4-add6d36a409c" (UID: "6d35c973-cc54-443b-b0a4-add6d36a409c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:30.099092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.099011 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-podinfo\") pod \"6d35c973-cc54-443b-b0a4-add6d36a409c\" (UID: \"6d35c973-cc54-443b-b0a4-add6d36a409c\") " Oct 30 22:51:30.099344 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.099108 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-data" pod "6d35c973-cc54-443b-b0a4-add6d36a409c" (UID: "6d35c973-cc54-443b-b0a4-add6d36a409c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:30.099344 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.099293 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d35c973-cc54-443b-b0a4-add6d36a409c-istiod-ca-cert" pod "6d35c973-cc54-443b-b0a4-add6d36a409c" (UID: "6d35c973-cc54-443b-b0a4-add6d36a409c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:30.099344 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.099272 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-certs" pod "6d35c973-cc54-443b-b0a4-add6d36a409c" (UID: "6d35c973-cc54-443b-b0a4-add6d36a409c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:30.099480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.099347 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-socket" pod "6d35c973-cc54-443b-b0a4-add6d36a409c" (UID: "6d35c973-cc54-443b-b0a4-add6d36a409c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:30.099527 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.099491 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.099527 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.099511 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.099801 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.099525 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d35c973-cc54-443b-b0a4-add6d36a409c-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.099801 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.099538 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.099801 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.099551 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.101304 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.101278 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-token" pod "6d35c973-cc54-443b-b0a4-add6d36a409c" (UID: "6d35c973-cc54-443b-b0a4-add6d36a409c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:30.101423 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.101331 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-podinfo" pod "6d35c973-cc54-443b-b0a4-add6d36a409c" (UID: "6d35c973-cc54-443b-b0a4-add6d36a409c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:51:30.101488 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.101454 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-envoy" pod "6d35c973-cc54-443b-b0a4-add6d36a409c" (UID: "6d35c973-cc54-443b-b0a4-add6d36a409c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:30.101541 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.101514 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-kube-api-access-86hcd" pod "6d35c973-cc54-443b-b0a4-add6d36a409c" (UID: "6d35c973-cc54-443b-b0a4-add6d36a409c"). InnerVolumeSpecName "kube-api-access-86hcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:30.200271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.200251 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86hcd\" (UniqueName: \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-kube-api-access-86hcd\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.200271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.200271 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.200594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.200281 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.200594 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.200291 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d35c973-cc54-443b-b0a4-add6d36a409c-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.513631 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.513600 4859 generic.go:356] "Generic (PLEG): container finished" podID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerID="1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4" exitCode=0 Oct 30 22:51:30.513807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.513651 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" Oct 30 22:51:30.513807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.513679 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" event={"ID":"6d35c973-cc54-443b-b0a4-add6d36a409c","Type":"ContainerDied","Data":"1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4"} Oct 30 22:51:30.513807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.513731 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck" event={"ID":"6d35c973-cc54-443b-b0a4-add6d36a409c","Type":"ContainerDied","Data":"e789601483b1612ad34077c43b56e58d7c739eb0ab1ddf94d2b22cc14d612b74"} Oct 30 22:51:30.513807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.513752 4859 scope.go:122] "RemoveContainer" containerID="1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4" Oct 30 22:51:30.524976 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.524959 4859 scope.go:122] "RemoveContainer" containerID="ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad" Oct 30 22:51:30.534874 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.534852 4859 scope.go:122] "RemoveContainer" containerID="1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4" Oct 30 22:51:30.535153 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:51:30.535135 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4\": container with ID starting with 1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4 not found: ID does not exist" containerID="1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4" Oct 30 22:51:30.535219 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.535167 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4"} err="failed to get container status \"1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4\": rpc error: code = NotFound desc = could not find container \"1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4\": container with ID starting with 1b003677bc8070d4e064a4319f00c7d0552a5c508f55487e65b8c96b6df9c6c4 not found: ID does not exist" Oct 30 22:51:30.535219 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.535186 4859 scope.go:122] "RemoveContainer" containerID="ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad" Oct 30 22:51:30.535440 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:51:30.535420 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad\": container with ID starting with ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad not found: ID does not exist" containerID="ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad" Oct 30 22:51:30.535506 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.535451 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad"} err="failed to get container status \"ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad\": rpc error: code = NotFound desc = could not find container \"ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad\": container with ID starting with ed56e839f790a07b5b82acab8782b2dd8e364cfce0591bc2e00e953a8d0554ad not found: ID does not exist" Oct 30 22:51:30.676914 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.676873 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck"] Oct 30 22:51:30.681903 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.681870 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69058c-openshift-default-848df6554f-pcwck"] Oct 30 22:51:30.744469 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:30.744440 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" path="/var/lib/kubelet/pods/6d35c973-cc54-443b-b0a4-add6d36a409c/volumes" Oct 30 22:51:34.869583 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:34.869550 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw"] Oct 30 22:51:34.979107 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:34.979073 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn"] Oct 30 22:51:37.337318 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:37.337288 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n"] Oct 30 22:51:37.451528 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:37.451497 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc"] Oct 30 22:51:39.870625 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:39.870583 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" podUID="1f3d68b6-bfaa-4994-9ace-6eedae9df25f" containerName="payload-processing" containerID="cri-o://4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa" gracePeriod=25 Oct 30 22:51:39.876714 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:51:39.876683 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:51:39.980316 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:39.980269 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" podUID="998baa49-be51-4cd6-95d3-b7e6288f974e" containerName="payload-pre-processing" containerID="cri-o://7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd" gracePeriod=25 Oct 30 22:51:42.338579 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.338530 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" podUID="3db3fd91-9283-4244-966d-4ac6378ce512" containerName="payload-processing" containerID="cri-o://1774386f8bfcb0ab71fd7a30de901c566c201cb4626a4e3bd548b0ad338ecb8c" gracePeriod=25 Oct 30 22:51:42.452507 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.452460 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" podUID="59fc40d6-745b-42b8-9918-cb6ae52ff6aa" containerName="payload-pre-processing" containerID="cri-o://57b7e07abf9e1c5f7a9aec72c7fc81b772bc6b12ae00686541aa44fc4ea29b42" gracePeriod=25 Oct 30 22:51:42.603643 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.603370 4859 generic.go:356] "Generic (PLEG): container finished" podID="3db3fd91-9283-4244-966d-4ac6378ce512" containerID="1774386f8bfcb0ab71fd7a30de901c566c201cb4626a4e3bd548b0ad338ecb8c" exitCode=0 Oct 30 22:51:42.603643 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.603442 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" event={"ID":"3db3fd91-9283-4244-966d-4ac6378ce512","Type":"ContainerDied","Data":"1774386f8bfcb0ab71fd7a30de901c566c201cb4626a4e3bd548b0ad338ecb8c"} Oct 30 22:51:42.606520 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.606403 4859 generic.go:356] "Generic (PLEG): container finished" podID="59fc40d6-745b-42b8-9918-cb6ae52ff6aa" containerID="57b7e07abf9e1c5f7a9aec72c7fc81b772bc6b12ae00686541aa44fc4ea29b42" exitCode=0 Oct 30 22:51:42.606520 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.606456 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" event={"ID":"59fc40d6-745b-42b8-9918-cb6ae52ff6aa","Type":"ContainerDied","Data":"57b7e07abf9e1c5f7a9aec72c7fc81b772bc6b12ae00686541aa44fc4ea29b42"} Oct 30 22:51:42.627198 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.627174 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 22:51:42.715689 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.715665 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 22:51:42.748215 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.748191 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db3fd91-9283-4244-966d-4ac6378ce512-kube-api-access-m2hq6\" (UniqueName: \"kubernetes.io/projected/3db3fd91-9283-4244-966d-4ac6378ce512-kube-api-access-m2hq6\") pod \"3db3fd91-9283-4244-966d-4ac6378ce512\" (UID: \"3db3fd91-9283-4244-966d-4ac6378ce512\") " Oct 30 22:51:42.748376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.748295 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db3fd91-9283-4244-966d-4ac6378ce512-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db3fd91-9283-4244-966d-4ac6378ce512-plugins-config-volume\") pod \"3db3fd91-9283-4244-966d-4ac6378ce512\" (UID: \"3db3fd91-9283-4244-966d-4ac6378ce512\") " Oct 30 22:51:42.748706 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.748679 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db3fd91-9283-4244-966d-4ac6378ce512-plugins-config-volume" pod "3db3fd91-9283-4244-966d-4ac6378ce512" (UID: "3db3fd91-9283-4244-966d-4ac6378ce512"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:42.750577 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.750556 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db3fd91-9283-4244-966d-4ac6378ce512-kube-api-access-m2hq6" pod "3db3fd91-9283-4244-966d-4ac6378ce512" (UID: "3db3fd91-9283-4244-966d-4ac6378ce512"). InnerVolumeSpecName "kube-api-access-m2hq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:42.849337 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.849308 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-kube-api-access-wjbsf\" (UniqueName: \"kubernetes.io/projected/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-kube-api-access-wjbsf\") pod \"59fc40d6-745b-42b8-9918-cb6ae52ff6aa\" (UID: \"59fc40d6-745b-42b8-9918-cb6ae52ff6aa\") " Oct 30 22:51:42.849529 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.849511 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-plugins-config-volume\") pod \"59fc40d6-745b-42b8-9918-cb6ae52ff6aa\" (UID: \"59fc40d6-745b-42b8-9918-cb6ae52ff6aa\") " Oct 30 22:51:42.849873 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.849845 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2hq6\" (UniqueName: \"kubernetes.io/projected/3db3fd91-9283-4244-966d-4ac6378ce512-kube-api-access-m2hq6\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:42.849873 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.849864 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db3fd91-9283-4244-966d-4ac6378ce512-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:42.850079 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.849909 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-plugins-config-volume" pod "59fc40d6-745b-42b8-9918-cb6ae52ff6aa" (UID: "59fc40d6-745b-42b8-9918-cb6ae52ff6aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:42.851411 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.851392 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-kube-api-access-wjbsf" pod "59fc40d6-745b-42b8-9918-cb6ae52ff6aa" (UID: "59fc40d6-745b-42b8-9918-cb6ae52ff6aa"). InnerVolumeSpecName "kube-api-access-wjbsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:42.951070 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.950999 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjbsf\" (UniqueName: \"kubernetes.io/projected/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-kube-api-access-wjbsf\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:42.951070 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:42.951034 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fc40d6-745b-42b8-9918-cb6ae52ff6aa-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:43.614595 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.614498 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" event={"ID":"3db3fd91-9283-4244-966d-4ac6378ce512","Type":"ContainerDied","Data":"109dc6553ac902c3b2ddcb3f3f87fb5694e13d124c7547f4544c06fd324613ff"} Oct 30 22:51:43.614595 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.614521 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n" Oct 30 22:51:43.614595 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.614565 4859 scope.go:122] "RemoveContainer" containerID="1774386f8bfcb0ab71fd7a30de901c566c201cb4626a4e3bd548b0ad338ecb8c" Oct 30 22:51:43.616210 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.616181 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" event={"ID":"59fc40d6-745b-42b8-9918-cb6ae52ff6aa","Type":"ContainerDied","Data":"ffd92b0cac3a789427c8ca2ff9b53704fabd31edee3783a0a5fa1a7b4203758e"} Oct 30 22:51:43.616333 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.616249 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc" Oct 30 22:51:43.626652 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.626631 4859 scope.go:122] "RemoveContainer" containerID="57b7e07abf9e1c5f7a9aec72c7fc81b772bc6b12ae00686541aa44fc4ea29b42" Oct 30 22:51:43.633681 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.633654 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" podUID="998baa49-be51-4cd6-95d3-b7e6288f974e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:51:43.633787 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.633654 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" podUID="1f3d68b6-bfaa-4994-9ace-6eedae9df25f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:51:43.780752 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.780679 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc"] Oct 30 22:51:43.786359 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.786323 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d2a3e0-6ffb8c767f-792tc"] Oct 30 22:51:43.891695 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.891618 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n"] Oct 30 22:51:43.896783 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:43.896759 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d2a3e0-5c54bc4fcc-9n25n"] Oct 30 22:51:44.740998 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:44.740968 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db3fd91-9283-4244-966d-4ac6378ce512" path="/var/lib/kubelet/pods/3db3fd91-9283-4244-966d-4ac6378ce512/volumes" Oct 30 22:51:44.741371 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:44.741322 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59fc40d6-745b-42b8-9918-cb6ae52ff6aa" path="/var/lib/kubelet/pods/59fc40d6-745b-42b8-9918-cb6ae52ff6aa/volumes" Oct 30 22:51:53.633043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:53.633006 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" podUID="998baa49-be51-4cd6-95d3-b7e6288f974e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:51:53.633531 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:53.633061 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" podUID="1f3d68b6-bfaa-4994-9ace-6eedae9df25f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:51:54.247719 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:51:54.247681 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:52:00.122210 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.122180 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 22:52:00.158607 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.158571 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-kube-api-access-m6m9d\" (UniqueName: \"kubernetes.io/projected/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-kube-api-access-m6m9d\") pod \"1f3d68b6-bfaa-4994-9ace-6eedae9df25f\" (UID: \"1f3d68b6-bfaa-4994-9ace-6eedae9df25f\") " Oct 30 22:52:00.158807 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.158681 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-plugins-config-volume\") pod \"1f3d68b6-bfaa-4994-9ace-6eedae9df25f\" (UID: \"1f3d68b6-bfaa-4994-9ace-6eedae9df25f\") " Oct 30 22:52:00.159342 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.159309 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-plugins-config-volume" pod "1f3d68b6-bfaa-4994-9ace-6eedae9df25f" (UID: "1f3d68b6-bfaa-4994-9ace-6eedae9df25f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:00.161995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.161967 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-kube-api-access-m6m9d" pod "1f3d68b6-bfaa-4994-9ace-6eedae9df25f" (UID: "1f3d68b6-bfaa-4994-9ace-6eedae9df25f"). InnerVolumeSpecName "kube-api-access-m6m9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:00.225164 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.225136 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 22:52:00.260050 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.260017 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998baa49-be51-4cd6-95d3-b7e6288f974e-kube-api-access-zbxkg\" (UniqueName: \"kubernetes.io/projected/998baa49-be51-4cd6-95d3-b7e6288f974e-kube-api-access-zbxkg\") pod \"998baa49-be51-4cd6-95d3-b7e6288f974e\" (UID: \"998baa49-be51-4cd6-95d3-b7e6288f974e\") " Oct 30 22:52:00.260248 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.260121 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998baa49-be51-4cd6-95d3-b7e6288f974e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998baa49-be51-4cd6-95d3-b7e6288f974e-plugins-config-volume\") pod \"998baa49-be51-4cd6-95d3-b7e6288f974e\" (UID: \"998baa49-be51-4cd6-95d3-b7e6288f974e\") " Oct 30 22:52:00.260546 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.260521 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998baa49-be51-4cd6-95d3-b7e6288f974e-plugins-config-volume" pod "998baa49-be51-4cd6-95d3-b7e6288f974e" (UID: "998baa49-be51-4cd6-95d3-b7e6288f974e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:00.260687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.260595 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6m9d\" (UniqueName: \"kubernetes.io/projected/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-kube-api-access-m6m9d\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.260687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.260625 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3d68b6-bfaa-4994-9ace-6eedae9df25f-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.262393 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.262373 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998baa49-be51-4cd6-95d3-b7e6288f974e-kube-api-access-zbxkg" pod "998baa49-be51-4cd6-95d3-b7e6288f974e" (UID: "998baa49-be51-4cd6-95d3-b7e6288f974e"). InnerVolumeSpecName "kube-api-access-zbxkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:00.361416 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.361384 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbxkg\" (UniqueName: \"kubernetes.io/projected/998baa49-be51-4cd6-95d3-b7e6288f974e-kube-api-access-zbxkg\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.361416 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.361417 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998baa49-be51-4cd6-95d3-b7e6288f974e-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.730434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.730395 4859 generic.go:356] "Generic (PLEG): container finished" podID="1f3d68b6-bfaa-4994-9ace-6eedae9df25f" containerID="4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa" exitCode=0 Oct 30 22:52:00.730621 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.730441 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" Oct 30 22:52:00.730621 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.730473 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" event={"ID":"1f3d68b6-bfaa-4994-9ace-6eedae9df25f","Type":"ContainerDied","Data":"4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa"} Oct 30 22:52:00.730621 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.730522 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw" event={"ID":"1f3d68b6-bfaa-4994-9ace-6eedae9df25f","Type":"ContainerDied","Data":"6d277185bab64a75e637b0e9c17b4ee0aeabe54cd4e13579337f90883fa02380"} Oct 30 22:52:00.730621 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.730548 4859 scope.go:122] "RemoveContainer" containerID="4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa" Oct 30 22:52:00.731919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.731895 4859 generic.go:356] "Generic (PLEG): container finished" podID="998baa49-be51-4cd6-95d3-b7e6288f974e" containerID="7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd" exitCode=0 Oct 30 22:52:00.732046 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.731940 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" Oct 30 22:52:00.732046 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.731991 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" event={"ID":"998baa49-be51-4cd6-95d3-b7e6288f974e","Type":"ContainerDied","Data":"7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd"} Oct 30 22:52:00.732147 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.732047 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn" event={"ID":"998baa49-be51-4cd6-95d3-b7e6288f974e","Type":"ContainerDied","Data":"cb1c654dc48bcb5dc0f8174cbbf54e4d6c060af16bc31528636434f5e9b8bdc0"} Oct 30 22:52:00.741682 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.741662 4859 scope.go:122] "RemoveContainer" containerID="4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa" Oct 30 22:52:00.741979 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:00.741956 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa\": container with ID starting with 4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa not found: ID does not exist" containerID="4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa" Oct 30 22:52:00.742094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.742070 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa"} err="failed to get container status \"4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa\": rpc error: code = NotFound desc = could not find container \"4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa\": container with ID starting with 4809f01dc6bcffe5b0d6898e7a4f42655dc050c33284b97bc9247b4c123005aa not found: ID does not exist" Oct 30 22:52:00.742201 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.742097 4859 scope.go:122] "RemoveContainer" containerID="7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd" Oct 30 22:52:00.752639 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.752623 4859 scope.go:122] "RemoveContainer" containerID="7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd" Oct 30 22:52:00.752882 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:00.752864 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd\": container with ID starting with 7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd not found: ID does not exist" containerID="7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd" Oct 30 22:52:00.752931 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.752889 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd"} err="failed to get container status \"7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd\": rpc error: code = NotFound desc = could not find container \"7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd\": container with ID starting with 7ce551f23b74bf201870affbe4c452088640892dcf23f80bbda2bce25002c7cd not found: ID does not exist" Oct 30 22:52:00.894093 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.894062 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw"] Oct 30 22:52:00.943255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:00.943218 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1c3eef-579584b4c9-g29mw"] Oct 30 22:52:01.052138 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:01.052064 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn"] Oct 30 22:52:01.057328 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:01.057301 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1c3eef-68464d9f59-l9kkn"] Oct 30 22:52:02.370907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.370868 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl"] Oct 30 22:52:02.371409 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.371085 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" podUID="964b48c3-e385-4359-9b3f-91df3e092b08" containerName="maas-api" containerID="cri-o://261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc" gracePeriod=30 Oct 30 22:52:02.654114 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.654085 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:52:02.697883 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.697853 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964b48c3-e385-4359-9b3f-91df3e092b08-kube-api-access-mdwzm\" (UniqueName: \"kubernetes.io/projected/964b48c3-e385-4359-9b3f-91df3e092b08-kube-api-access-mdwzm\") pod \"964b48c3-e385-4359-9b3f-91df3e092b08\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " Oct 30 22:52:02.698092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.697909 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-maas-api-tls\") pod \"964b48c3-e385-4359-9b3f-91df3e092b08\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " Oct 30 22:52:02.698092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.698011 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964b48c3-e385-4359-9b3f-91df3e092b08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/964b48c3-e385-4359-9b3f-91df3e092b08-openshift-service-ca\") pod \"964b48c3-e385-4359-9b3f-91df3e092b08\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " Oct 30 22:52:02.698092 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.698063 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-metrics-certs\") pod \"964b48c3-e385-4359-9b3f-91df3e092b08\" (UID: \"964b48c3-e385-4359-9b3f-91df3e092b08\") " Oct 30 22:52:02.698569 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.698535 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964b48c3-e385-4359-9b3f-91df3e092b08-openshift-service-ca" pod "964b48c3-e385-4359-9b3f-91df3e092b08" (UID: "964b48c3-e385-4359-9b3f-91df3e092b08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:02.700499 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.700458 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-maas-api-tls" pod "964b48c3-e385-4359-9b3f-91df3e092b08" (UID: "964b48c3-e385-4359-9b3f-91df3e092b08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:02.700617 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.700551 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-metrics-certs" pod "964b48c3-e385-4359-9b3f-91df3e092b08" (UID: "964b48c3-e385-4359-9b3f-91df3e092b08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:02.700617 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.700578 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964b48c3-e385-4359-9b3f-91df3e092b08-kube-api-access-mdwzm" pod "964b48c3-e385-4359-9b3f-91df3e092b08" (UID: "964b48c3-e385-4359-9b3f-91df3e092b08"). InnerVolumeSpecName "kube-api-access-mdwzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:02.776389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.776128 4859 generic.go:356] "Generic (PLEG): container finished" podID="964b48c3-e385-4359-9b3f-91df3e092b08" containerID="261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc" exitCode=0 Oct 30 22:52:02.780773 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.780733 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" Oct 30 22:52:02.785995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.783644 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3d68b6-bfaa-4994-9ace-6eedae9df25f" path="/var/lib/kubelet/pods/1f3d68b6-bfaa-4994-9ace-6eedae9df25f/volumes" Oct 30 22:52:02.785995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.784076 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998baa49-be51-4cd6-95d3-b7e6288f974e" path="/var/lib/kubelet/pods/998baa49-be51-4cd6-95d3-b7e6288f974e/volumes" Oct 30 22:52:02.785995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.784471 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" event={"ID":"964b48c3-e385-4359-9b3f-91df3e092b08","Type":"ContainerDied","Data":"261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc"} Oct 30 22:52:02.785995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.784505 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl" event={"ID":"964b48c3-e385-4359-9b3f-91df3e092b08","Type":"ContainerDied","Data":"086e0f2c43ceebbe9612d8a3141dbdb7aacbf55aabae05a4702fba7ed6cdf368"} Oct 30 22:52:02.785995 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.784528 4859 scope.go:122] "RemoveContainer" containerID="261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc" Oct 30 22:52:02.800168 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.800133 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdwzm\" (UniqueName: \"kubernetes.io/projected/964b48c3-e385-4359-9b3f-91df3e092b08-kube-api-access-mdwzm\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:02.800168 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.800161 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:02.800336 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.800176 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/964b48c3-e385-4359-9b3f-91df3e092b08-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:02.800336 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.800189 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/964b48c3-e385-4359-9b3f-91df3e092b08-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:02.800743 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.800724 4859 scope.go:122] "RemoveContainer" containerID="261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc" Oct 30 22:52:02.801205 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:02.801175 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc\": container with ID starting with 261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc not found: ID does not exist" containerID="261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc" Oct 30 22:52:02.801278 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.801215 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc"} err="failed to get container status \"261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc\": rpc error: code = NotFound desc = could not find container \"261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc\": container with ID starting with 261dd4001c0f1a8b39e9a255588827c08389cc7606c5a77909aaa29cfb3d04fc not found: ID does not exist" Oct 30 22:52:02.949448 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.949365 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl"] Oct 30 22:52:02.955128 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:02.955096 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a0116e-859dffc569-hnjnl"] Oct 30 22:52:04.741053 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:04.741025 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964b48c3-e385-4359-9b3f-91df3e092b08" path="/var/lib/kubelet/pods/964b48c3-e385-4359-9b3f-91df3e092b08/volumes" Oct 30 22:52:09.879897 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:09.879852 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:52:11.373147 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:11.373111 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg"] Oct 30 22:52:11.373623 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:11.373362 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" containerID="cri-o://ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13" gracePeriod=30 Oct 30 22:52:16.659596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.659543 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:52:16.769157 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769129 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-credential-socket\") pod \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " Oct 30 22:52:16.769323 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769182 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-socket\") pod \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " Oct 30 22:52:16.769323 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769300 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-token\" (UniqueName: \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-token\") pod \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " Oct 30 22:52:16.769443 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769337 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-certs\") pod \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " Oct 30 22:52:16.769443 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769368 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-podinfo\") pod \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " Oct 30 22:52:16.769443 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769412 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-kube-api-access-tmqvr\" (UniqueName: \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-kube-api-access-tmqvr\") pod \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " Oct 30 22:52:16.769443 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769426 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-credential-socket" pod "e3efccd2-b4f7-47ad-ac24-326f5246aac6" (UID: "e3efccd2-b4f7-47ad-ac24-326f5246aac6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:16.769659 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769456 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istiod-ca-cert\") pod \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " Oct 30 22:52:16.769659 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769512 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-data\") pod \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " Oct 30 22:52:16.769659 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769545 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-envoy\") pod \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\" (UID: \"e3efccd2-b4f7-47ad-ac24-326f5246aac6\") " Oct 30 22:52:16.769822 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769685 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-certs" pod "e3efccd2-b4f7-47ad-ac24-326f5246aac6" (UID: "e3efccd2-b4f7-47ad-ac24-326f5246aac6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:16.769999 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769934 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-data" pod "e3efccd2-b4f7-47ad-ac24-326f5246aac6" (UID: "e3efccd2-b4f7-47ad-ac24-326f5246aac6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:16.769999 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.769989 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:16.770193 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.770011 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:16.770257 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.770192 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istiod-ca-cert" pod "e3efccd2-b4f7-47ad-ac24-326f5246aac6" (UID: "e3efccd2-b4f7-47ad-ac24-326f5246aac6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:16.770523 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.770500 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-socket" pod "e3efccd2-b4f7-47ad-ac24-326f5246aac6" (UID: "e3efccd2-b4f7-47ad-ac24-326f5246aac6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:16.771719 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.771686 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-token" pod "e3efccd2-b4f7-47ad-ac24-326f5246aac6" (UID: "e3efccd2-b4f7-47ad-ac24-326f5246aac6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:16.771833 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.771810 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-kube-api-access-tmqvr" pod "e3efccd2-b4f7-47ad-ac24-326f5246aac6" (UID: "e3efccd2-b4f7-47ad-ac24-326f5246aac6"). InnerVolumeSpecName "kube-api-access-tmqvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:16.772308 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.772286 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-podinfo" pod "e3efccd2-b4f7-47ad-ac24-326f5246aac6" (UID: "e3efccd2-b4f7-47ad-ac24-326f5246aac6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:52:16.772471 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.772454 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-envoy" pod "e3efccd2-b4f7-47ad-ac24-326f5246aac6" (UID: "e3efccd2-b4f7-47ad-ac24-326f5246aac6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:16.871547 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.871517 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:16.871547 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.871541 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:16.871547 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.871550 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmqvr\" (UniqueName: \"kubernetes.io/projected/e3efccd2-b4f7-47ad-ac24-326f5246aac6-kube-api-access-tmqvr\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:16.871754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.871562 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:16.871754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.871570 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:16.871754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.871578 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:16.871754 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.871586 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3efccd2-b4f7-47ad-ac24-326f5246aac6-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:16.883472 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.883443 4859 generic.go:356] "Generic (PLEG): container finished" podID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerID="ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13" exitCode=0 Oct 30 22:52:16.883617 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.883492 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" Oct 30 22:52:16.883617 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.883526 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" event={"ID":"e3efccd2-b4f7-47ad-ac24-326f5246aac6","Type":"ContainerDied","Data":"ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13"} Oct 30 22:52:16.883617 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.883581 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg" event={"ID":"e3efccd2-b4f7-47ad-ac24-326f5246aac6","Type":"ContainerDied","Data":"3c5b65c94a5355756377af5f587fc7a3d211fb66f3fa2d53e55056b8d30366f3"} Oct 30 22:52:16.883617 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.883608 4859 scope.go:122] "RemoveContainer" containerID="ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13" Oct 30 22:52:16.894004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.893985 4859 scope.go:122] "RemoveContainer" containerID="efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98" Oct 30 22:52:16.902565 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.902549 4859 scope.go:122] "RemoveContainer" containerID="ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13" Oct 30 22:52:16.902797 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:16.902779 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13\": container with ID starting with ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13 not found: ID does not exist" containerID="ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13" Oct 30 22:52:16.902867 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.902803 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13"} err="failed to get container status \"ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13\": rpc error: code = NotFound desc = could not find container \"ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13\": container with ID starting with ecdbc02ec88b01f03045ef173c313b14a2ca3fa32784d47390bc9d97b2c5ef13 not found: ID does not exist" Oct 30 22:52:16.902867 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.902819 4859 scope.go:122] "RemoveContainer" containerID="efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98" Oct 30 22:52:16.903063 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:16.903044 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98\": container with ID starting with efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98 not found: ID does not exist" containerID="efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98" Oct 30 22:52:16.903130 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:16.903067 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98"} err="failed to get container status \"efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98\": rpc error: code = NotFound desc = could not find container \"efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98\": container with ID starting with efdaf21b6cd78a815284551c9ff260442c76005e4187cfb28502570692504c98 not found: ID does not exist" Oct 30 22:52:17.054773 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:17.054696 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg"] Oct 30 22:52:17.058561 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:17.058528 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a0116e-openshift-default-d69c5cbd7-zd8mg"] Oct 30 22:52:18.742455 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:18.742421 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" path="/var/lib/kubelet/pods/e3efccd2-b4f7-47ad-ac24-326f5246aac6/volumes" Oct 30 22:52:23.527369 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:23.527342 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:52:23.535484 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:23.535460 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:52:23.552284 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:23.552264 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:52:23.559539 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:23.559520 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:52:37.992444 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:37.992411 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms"] Oct 30 22:52:37.992910 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:37.992659 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" containerID="cri-o://deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97" gracePeriod=30 Oct 30 22:52:37.998671 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:37.998638 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g"] Oct 30 22:52:37.998926 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:37.998878 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" podUID="4f8242e9-9b3b-429a-ad38-09dad5c6ef30" containerName="maas-api" containerID="cri-o://eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf" gracePeriod=30 Oct 30 22:52:38.163924 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.163891 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:52:38.262227 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.261912 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:52:38.262227 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.261998 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:52:38.327648 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.327624 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 22:52:38.450353 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.450315 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-kube-api-access-2q9x9\" (UniqueName: \"kubernetes.io/projected/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-kube-api-access-2q9x9\") pod \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " Oct 30 22:52:38.450544 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.450450 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-maas-api-tls\") pod \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " Oct 30 22:52:38.450544 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.450501 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-metrics-certs\") pod \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " Oct 30 22:52:38.450544 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.450539 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-openshift-service-ca\") pod \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\" (UID: \"4f8242e9-9b3b-429a-ad38-09dad5c6ef30\") " Oct 30 22:52:38.450936 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.450906 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-openshift-service-ca" pod "4f8242e9-9b3b-429a-ad38-09dad5c6ef30" (UID: "4f8242e9-9b3b-429a-ad38-09dad5c6ef30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:38.453004 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.452968 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-kube-api-access-2q9x9" pod "4f8242e9-9b3b-429a-ad38-09dad5c6ef30" (UID: "4f8242e9-9b3b-429a-ad38-09dad5c6ef30"). InnerVolumeSpecName "kube-api-access-2q9x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:38.453121 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.453046 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-maas-api-tls" pod "4f8242e9-9b3b-429a-ad38-09dad5c6ef30" (UID: "4f8242e9-9b3b-429a-ad38-09dad5c6ef30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:38.453121 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.453061 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-metrics-certs" pod "4f8242e9-9b3b-429a-ad38-09dad5c6ef30" (UID: "4f8242e9-9b3b-429a-ad38-09dad5c6ef30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:38.552665 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.552574 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:38.552665 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.552613 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:38.552665 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.552627 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q9x9\" (UniqueName: \"kubernetes.io/projected/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-kube-api-access-2q9x9\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:38.552665 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:38.552643 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f8242e9-9b3b-429a-ad38-09dad5c6ef30-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:39.041386 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:39.041333 4859 generic.go:356] "Generic (PLEG): container finished" podID="4f8242e9-9b3b-429a-ad38-09dad5c6ef30" containerID="eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf" exitCode=0 Oct 30 22:52:39.041805 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:39.041404 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" Oct 30 22:52:39.041805 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:39.041423 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" event={"ID":"4f8242e9-9b3b-429a-ad38-09dad5c6ef30","Type":"ContainerDied","Data":"eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf"} Oct 30 22:52:39.041805 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:39.041492 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g" event={"ID":"4f8242e9-9b3b-429a-ad38-09dad5c6ef30","Type":"ContainerDied","Data":"fa33576a534978db7e20415a1ac34abb54ca618cb548eb85f043c0cf7529924b"} Oct 30 22:52:39.041805 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:39.041522 4859 scope.go:122] "RemoveContainer" containerID="eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf" Oct 30 22:52:39.051700 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:39.051682 4859 scope.go:122] "RemoveContainer" containerID="eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf" Oct 30 22:52:39.052014 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:39.051983 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf\": container with ID starting with eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf not found: ID does not exist" containerID="eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf" Oct 30 22:52:39.052014 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:39.052007 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf"} err="failed to get container status \"eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf\": rpc error: code = NotFound desc = could not find container \"eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf\": container with ID starting with eef2be65a5d070c25e4c47be30575f7f28c0d84a95a9e1a92a0649568b1305cf not found: ID does not exist" Oct 30 22:52:39.206140 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:39.206105 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g"] Oct 30 22:52:39.208549 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:39.208529 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1c3eef-65b8bcbd8b-fkl9g"] Oct 30 22:52:39.884401 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:39.884368 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:52:40.744905 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:40.744868 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8242e9-9b3b-429a-ad38-09dad5c6ef30" path="/var/lib/kubelet/pods/4f8242e9-9b3b-429a-ad38-09dad5c6ef30/volumes" Oct 30 22:52:43.251880 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.251852 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:52:43.394486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.394445 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565"] Oct 30 22:52:43.407701 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.407670 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-token\" (UniqueName: \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-token\") pod \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " Oct 30 22:52:43.407910 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.407748 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-podinfo\") pod \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " Oct 30 22:52:43.407910 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.407790 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-envoy\") pod \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " Oct 30 22:52:43.407910 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.407870 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-socket\") pod \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.407924 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istiod-ca-cert\") pod \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.407992 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-credential-socket\") pod \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.408067 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-data\") pod \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.408151 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-certs\") pod \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.408222 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-kube-api-access-7mqwj\" (UniqueName: \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-kube-api-access-7mqwj\") pod \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\" (UID: \"2c1491e9-ac51-4e0b-828a-fe8984ff6282\") " Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.408833 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-socket" pod "2c1491e9-ac51-4e0b-828a-fe8984ff6282" (UID: "2c1491e9-ac51-4e0b-828a-fe8984ff6282"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.409229 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-credential-socket" pod "2c1491e9-ac51-4e0b-828a-fe8984ff6282" (UID: "2c1491e9-ac51-4e0b-828a-fe8984ff6282"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.409520 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istiod-ca-cert" pod "2c1491e9-ac51-4e0b-828a-fe8984ff6282" (UID: "2c1491e9-ac51-4e0b-828a-fe8984ff6282"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.409699 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-certs" pod "2c1491e9-ac51-4e0b-828a-fe8984ff6282" (UID: "2c1491e9-ac51-4e0b-828a-fe8984ff6282"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.411559 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-data" pod "2c1491e9-ac51-4e0b-828a-fe8984ff6282" (UID: "2c1491e9-ac51-4e0b-828a-fe8984ff6282"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:43.415011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.414316 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-kube-api-access-7mqwj" pod "2c1491e9-ac51-4e0b-828a-fe8984ff6282" (UID: "2c1491e9-ac51-4e0b-828a-fe8984ff6282"). InnerVolumeSpecName "kube-api-access-7mqwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:43.416123 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.415050 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-podinfo" pod "2c1491e9-ac51-4e0b-828a-fe8984ff6282" (UID: "2c1491e9-ac51-4e0b-828a-fe8984ff6282"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:52:43.416123 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.415130 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-token" pod "2c1491e9-ac51-4e0b-828a-fe8984ff6282" (UID: "2c1491e9-ac51-4e0b-828a-fe8984ff6282"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:43.416123 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.415179 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-envoy" pod "2c1491e9-ac51-4e0b-828a-fe8984ff6282" (UID: "2c1491e9-ac51-4e0b-828a-fe8984ff6282"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:43.509553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.509512 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:43.509553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.509553 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:43.509553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.509567 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:43.509865 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.509580 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mqwj\" (UniqueName: \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-kube-api-access-7mqwj\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:43.509865 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.509596 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:43.509865 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.509609 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:43.509865 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.509622 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:43.509865 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.509636 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1491e9-ac51-4e0b-828a-fe8984ff6282-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:43.509865 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:43.509649 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1491e9-ac51-4e0b-828a-fe8984ff6282-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:44.076055 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.076018 4859 generic.go:356] "Generic (PLEG): container finished" podID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerID="deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97" exitCode=0 Oct 30 22:52:44.076271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.076071 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" Oct 30 22:52:44.076271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.076108 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" event={"ID":"2c1491e9-ac51-4e0b-828a-fe8984ff6282","Type":"ContainerDied","Data":"deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97"} Oct 30 22:52:44.076271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.076162 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms" event={"ID":"2c1491e9-ac51-4e0b-828a-fe8984ff6282","Type":"ContainerDied","Data":"499b894763e73933e9abe9c67b23ca268ada15381e2bacb198948dcb0f49a67d"} Oct 30 22:52:44.076271 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.076191 4859 scope.go:122] "RemoveContainer" containerID="deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97" Oct 30 22:52:44.088483 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.088453 4859 scope.go:122] "RemoveContainer" containerID="732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa" Oct 30 22:52:44.098015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.097991 4859 scope.go:122] "RemoveContainer" containerID="deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97" Oct 30 22:52:44.098374 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:44.098340 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97\": container with ID starting with deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97 not found: ID does not exist" containerID="deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97" Oct 30 22:52:44.098497 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.098378 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97"} err="failed to get container status \"deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97\": rpc error: code = NotFound desc = could not find container \"deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97\": container with ID starting with deae662b17c0c6d97c86460ce9a510df9da84393e29b19f8df33329943d62e97 not found: ID does not exist" Oct 30 22:52:44.098497 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.098401 4859 scope.go:122] "RemoveContainer" containerID="732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa" Oct 30 22:52:44.098720 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:44.098697 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa\": container with ID starting with 732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa not found: ID does not exist" containerID="732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa" Oct 30 22:52:44.098794 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.098735 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa"} err="failed to get container status \"732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa\": rpc error: code = NotFound desc = could not find container \"732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa\": container with ID starting with 732fc56fc9ac60bc719eddd243bf15bd65e41bcfcf61adca50a820496afe5efa not found: ID does not exist" Oct 30 22:52:44.240366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.240332 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms"] Oct 30 22:52:44.242918 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.242897 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b1eb3f-openshift-default-699765bbd7-hp9ms"] Oct 30 22:52:44.741049 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:44.741013 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" path="/var/lib/kubelet/pods/2c1491e9-ac51-4e0b-828a-fe8984ff6282/volumes" Oct 30 22:52:58.422173 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.422129 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" podUID="ec42687d-7e39-4655-8873-8f790f18baf6" containerName="main" containerID="cri-o://7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea" gracePeriod=45 Oct 30 22:52:58.530795 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.530760 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" podUID="ec42687d-7e39-4655-8873-8f790f18baf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": dial tcp 10.132.0.102:8000: connect: connection refused" Oct 30 22:52:58.696674 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.696642 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:52:58.787434 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.787405 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-dshm\") pod \"ec42687d-7e39-4655-8873-8f790f18baf6\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " Oct 30 22:52:58.787613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.787446 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-home\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-home\") pod \"ec42687d-7e39-4655-8873-8f790f18baf6\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " Oct 30 22:52:58.787613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.787500 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec42687d-7e39-4655-8873-8f790f18baf6-kube-api-access-6qg28\" (UniqueName: \"kubernetes.io/projected/ec42687d-7e39-4655-8873-8f790f18baf6-kube-api-access-6qg28\") pod \"ec42687d-7e39-4655-8873-8f790f18baf6\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " Oct 30 22:52:58.787613 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.787535 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-model-cache\") pod \"ec42687d-7e39-4655-8873-8f790f18baf6\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " Oct 30 22:52:58.787824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.787681 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec42687d-7e39-4655-8873-8f790f18baf6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec42687d-7e39-4655-8873-8f790f18baf6-tls-certs\") pod \"ec42687d-7e39-4655-8873-8f790f18baf6\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " Oct 30 22:52:58.787824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.787716 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-tmp-dir\") pod \"ec42687d-7e39-4655-8873-8f790f18baf6\" (UID: \"ec42687d-7e39-4655-8873-8f790f18baf6\") " Oct 30 22:52:58.787824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.787778 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-home" pod "ec42687d-7e39-4655-8873-8f790f18baf6" (UID: "ec42687d-7e39-4655-8873-8f790f18baf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:58.788225 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.788061 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-tmp-dir" pod "ec42687d-7e39-4655-8873-8f790f18baf6" (UID: "ec42687d-7e39-4655-8873-8f790f18baf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:58.788225 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.788126 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-model-cache" pod "ec42687d-7e39-4655-8873-8f790f18baf6" (UID: "ec42687d-7e39-4655-8873-8f790f18baf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:58.788225 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.788182 4859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-home\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:58.788225 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.788200 4859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-tmp-dir\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:58.790036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.790012 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-dshm" pod "ec42687d-7e39-4655-8873-8f790f18baf6" (UID: "ec42687d-7e39-4655-8873-8f790f18baf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:58.790220 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.790200 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec42687d-7e39-4655-8873-8f790f18baf6-tls-certs" pod "ec42687d-7e39-4655-8873-8f790f18baf6" (UID: "ec42687d-7e39-4655-8873-8f790f18baf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:58.790359 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.790341 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec42687d-7e39-4655-8873-8f790f18baf6-kube-api-access-6qg28" pod "ec42687d-7e39-4655-8873-8f790f18baf6" (UID: "ec42687d-7e39-4655-8873-8f790f18baf6"). InnerVolumeSpecName "kube-api-access-6qg28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:58.889962 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.889804 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qg28\" (UniqueName: \"kubernetes.io/projected/ec42687d-7e39-4655-8873-8f790f18baf6-kube-api-access-6qg28\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:58.889962 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.889841 4859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-model-cache\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:58.889962 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.889855 4859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec42687d-7e39-4655-8873-8f790f18baf6-tls-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:58.889962 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:58.889868 4859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec42687d-7e39-4655-8873-8f790f18baf6-dshm\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:59.171388 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:59.171350 4859 generic.go:356] "Generic (PLEG): container finished" podID="ec42687d-7e39-4655-8873-8f790f18baf6" containerID="7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea" exitCode=0 Oct 30 22:52:59.171591 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:59.171405 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" Oct 30 22:52:59.171591 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:59.171423 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" event={"ID":"ec42687d-7e39-4655-8873-8f790f18baf6","Type":"ContainerDied","Data":"7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea"} Oct 30 22:52:59.171591 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:59.171459 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565" event={"ID":"ec42687d-7e39-4655-8873-8f790f18baf6","Type":"ContainerDied","Data":"c3afd91673672adb10e6ea568319692d923e4698d3029949a358ada5b2fe93aa"} Oct 30 22:52:59.171591 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:59.171474 4859 scope.go:122] "RemoveContainer" containerID="7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea" Oct 30 22:52:59.182428 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:59.182411 4859 scope.go:122] "RemoveContainer" containerID="7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea" Oct 30 22:52:59.182693 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:52:59.182676 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea\": container with ID starting with 7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea not found: ID does not exist" containerID="7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea" Oct 30 22:52:59.182756 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:59.182700 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea"} err="failed to get container status \"7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea\": rpc error: code = NotFound desc = could not find container \"7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea\": container with ID starting with 7c1ffe0026621c04eb32c6776039dd7326758363e741ae83ed668866bd2710ea not found: ID does not exist" Oct 30 22:52:59.333765 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:59.333738 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565"] Oct 30 22:52:59.336174 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:52:59.336155 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-2-w0-b8717e-kserve-866884dffb-8j565"] Oct 30 22:53:00.740146 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:00.740113 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec42687d-7e39-4655-8873-8f790f18baf6" path="/var/lib/kubelet/pods/ec42687d-7e39-4655-8873-8f790f18baf6/volumes" Oct 30 22:53:09.878299 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:53:09.878255 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:53:12.966821 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:12.966789 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:53:23.876427 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:23.876392 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655"] Oct 30 22:53:23.879045 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:23.876630 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" containerID="cri-o://30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92" gracePeriod=30 Oct 30 22:53:24.959919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:24.959886 4859 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:53:24.960307 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:24.959958 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:53:29.158522 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.158457 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:53:29.235027 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.234990 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-envoy\") pod \"fdb061db-828f-4cc8-aa1f-116685012e01\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " Oct 30 22:53:29.235216 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235038 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-istio-token\" (UniqueName: \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-istio-token\") pod \"fdb061db-828f-4cc8-aa1f-116685012e01\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " Oct 30 22:53:29.235216 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235089 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-data\") pod \"fdb061db-828f-4cc8-aa1f-116685012e01\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " Oct 30 22:53:29.235216 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235143 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-credential-socket\") pod \"fdb061db-828f-4cc8-aa1f-116685012e01\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " Oct 30 22:53:29.235216 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235179 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb061db-828f-4cc8-aa1f-116685012e01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb061db-828f-4cc8-aa1f-116685012e01-istiod-ca-cert\") pod \"fdb061db-828f-4cc8-aa1f-116685012e01\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " Oct 30 22:53:29.235422 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235246 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdb061db-828f-4cc8-aa1f-116685012e01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb061db-828f-4cc8-aa1f-116685012e01-istio-podinfo\") pod \"fdb061db-828f-4cc8-aa1f-116685012e01\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " Oct 30 22:53:29.235543 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235486 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-kube-api-access-mv986\" (UniqueName: \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-kube-api-access-mv986\") pod \"fdb061db-828f-4cc8-aa1f-116685012e01\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " Oct 30 22:53:29.235543 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235516 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-data" pod "fdb061db-828f-4cc8-aa1f-116685012e01" (UID: "fdb061db-828f-4cc8-aa1f-116685012e01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:29.235642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235534 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-credential-socket" pod "fdb061db-828f-4cc8-aa1f-116685012e01" (UID: "fdb061db-828f-4cc8-aa1f-116685012e01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:29.235642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235548 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-certs\") pod \"fdb061db-828f-4cc8-aa1f-116685012e01\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " Oct 30 22:53:29.235642 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235594 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-socket\") pod \"fdb061db-828f-4cc8-aa1f-116685012e01\" (UID: \"fdb061db-828f-4cc8-aa1f-116685012e01\") " Oct 30 22:53:29.235785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235596 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb061db-828f-4cc8-aa1f-116685012e01-istiod-ca-cert" pod "fdb061db-828f-4cc8-aa1f-116685012e01" (UID: "fdb061db-828f-4cc8-aa1f-116685012e01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:29.235851 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235793 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-socket" pod "fdb061db-828f-4cc8-aa1f-116685012e01" (UID: "fdb061db-828f-4cc8-aa1f-116685012e01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:29.235851 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.235836 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-certs" pod "fdb061db-828f-4cc8-aa1f-116685012e01" (UID: "fdb061db-828f-4cc8-aa1f-116685012e01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:29.236043 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.236027 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.236118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.236046 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb061db-828f-4cc8-aa1f-116685012e01-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.236118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.236059 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.236118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.236072 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.236118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.236084 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.237305 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.237282 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-istio-token" pod "fdb061db-828f-4cc8-aa1f-116685012e01" (UID: "fdb061db-828f-4cc8-aa1f-116685012e01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:29.237417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.237399 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-kube-api-access-mv986" pod "fdb061db-828f-4cc8-aa1f-116685012e01" (UID: "fdb061db-828f-4cc8-aa1f-116685012e01"). InnerVolumeSpecName "kube-api-access-mv986". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:29.237571 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.237555 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-envoy" pod "fdb061db-828f-4cc8-aa1f-116685012e01" (UID: "fdb061db-828f-4cc8-aa1f-116685012e01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:29.237799 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.237776 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdb061db-828f-4cc8-aa1f-116685012e01-istio-podinfo" pod "fdb061db-828f-4cc8-aa1f-116685012e01" (UID: "fdb061db-828f-4cc8-aa1f-116685012e01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:53:29.310715 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.310679 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp"] Oct 30 22:53:29.336958 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.336923 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.337132 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.336983 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb061db-828f-4cc8-aa1f-116685012e01-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.337132 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.336995 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv986\" (UniqueName: \"kubernetes.io/projected/fdb061db-828f-4cc8-aa1f-116685012e01-kube-api-access-mv986\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.337132 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.337005 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb061db-828f-4cc8-aa1f-116685012e01-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.369167 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.369131 4859 generic.go:356] "Generic (PLEG): container finished" podID="fdb061db-828f-4cc8-aa1f-116685012e01" containerID="30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92" exitCode=0 Oct 30 22:53:29.369352 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.369334 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" event={"ID":"fdb061db-828f-4cc8-aa1f-116685012e01","Type":"ContainerDied","Data":"30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92"} Oct 30 22:53:29.369398 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.369387 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" event={"ID":"fdb061db-828f-4cc8-aa1f-116685012e01","Type":"ContainerDied","Data":"c9516bbeb69022c479d322c8904622c0a9f1c1ff6d45c02188fc407d18437efc"} Oct 30 22:53:29.369452 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.369415 4859 scope.go:122] "RemoveContainer" containerID="30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92" Oct 30 22:53:29.369581 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.369570 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655" Oct 30 22:53:29.400252 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.400222 4859 scope.go:122] "RemoveContainer" containerID="9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f" Oct 30 22:53:29.409441 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.409416 4859 scope.go:122] "RemoveContainer" containerID="30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92" Oct 30 22:53:29.409771 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:53:29.409743 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92\": container with ID starting with 30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92 not found: ID does not exist" containerID="30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92" Oct 30 22:53:29.409886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.409783 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92"} err="failed to get container status \"30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92\": rpc error: code = NotFound desc = could not find container \"30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92\": container with ID starting with 30f8d9593e125e0a70b62eafdb18be536910fa36218b6d198c84d9e407751a92 not found: ID does not exist" Oct 30 22:53:29.409886 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.409810 4859 scope.go:122] "RemoveContainer" containerID="9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f" Oct 30 22:53:29.410126 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:53:29.410103 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f\": container with ID starting with 9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f not found: ID does not exist" containerID="9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f" Oct 30 22:53:29.410183 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.410136 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f"} err="failed to get container status \"9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f\": rpc error: code = NotFound desc = could not find container \"9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f\": container with ID starting with 9934514ff0ad04d799c29aba23ecfe77b4fdcd24f6f8fb80493886ae7211694f not found: ID does not exist" Oct 30 22:53:29.552121 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.552083 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655"] Oct 30 22:53:29.563176 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:29.563144 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6c788d-openshift-default-65f5f4d985-lh655"] Oct 30 22:53:30.536210 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:30.536172 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:53:30.740109 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:30.740075 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" path="/var/lib/kubelet/pods/fdb061db-828f-4cc8-aa1f-116685012e01/volumes" Oct 30 22:53:36.455221 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:36.455130 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn"] Oct 30 22:53:36.562228 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:36.562191 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq"] Oct 30 22:53:39.878939 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:53:39.878900 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:53:41.455798 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:41.455759 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" podUID="f838b548-a378-4880-a793-bcdefc06f68b" containerName="payload-processing" containerID="cri-o://8a166d3ff2c7af068e2bf7b2ec21c508cd82a0e29a7c9e3530f7e61cf79b7820" gracePeriod=25 Oct 30 22:53:41.563598 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:41.563555 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" podUID="2f093449-0fc4-40b1-a47b-2dfdcd448e07" containerName="payload-pre-processing" containerID="cri-o://8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85" gracePeriod=25 Oct 30 22:53:43.391153 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:43.391115 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" podUID="2f093449-0fc4-40b1-a47b-2dfdcd448e07" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:43.391586 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:43.391154 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" podUID="f838b548-a378-4880-a793-bcdefc06f68b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:44.330690 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.330652 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" containerID="cri-o://aa055bf1a7b368f8021c261d1c3ad6926c51afb6c9ade4eb9345a3f5f165dda3" gracePeriod=45 Oct 30 22:53:44.467719 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.467684 4859 generic.go:356] "Generic (PLEG): container finished" podID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerID="aa055bf1a7b368f8021c261d1c3ad6926c51afb6c9ade4eb9345a3f5f165dda3" exitCode=0 Oct 30 22:53:44.468058 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.467750 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" event={"ID":"47691dfc-77d7-47dc-9922-d439e2de1fe2","Type":"ContainerDied","Data":"aa055bf1a7b368f8021c261d1c3ad6926c51afb6c9ade4eb9345a3f5f165dda3"} Oct 30 22:53:44.590553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.590531 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 22:53:44.592668 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.592650 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-dshm\") pod \"47691dfc-77d7-47dc-9922-d439e2de1fe2\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " Oct 30 22:53:44.592724 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.592685 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-model-cache\") pod \"47691dfc-77d7-47dc-9922-d439e2de1fe2\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " Oct 30 22:53:44.592724 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.592708 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47691dfc-77d7-47dc-9922-d439e2de1fe2-kube-api-access-tdvr2\" (UniqueName: \"kubernetes.io/projected/47691dfc-77d7-47dc-9922-d439e2de1fe2-kube-api-access-tdvr2\") pod \"47691dfc-77d7-47dc-9922-d439e2de1fe2\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " Oct 30 22:53:44.592804 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.592768 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-tmp-dir\") pod \"47691dfc-77d7-47dc-9922-d439e2de1fe2\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " Oct 30 22:53:44.592858 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.592847 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-home\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-home\") pod \"47691dfc-77d7-47dc-9922-d439e2de1fe2\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " Oct 30 22:53:44.592913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.592873 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47691dfc-77d7-47dc-9922-d439e2de1fe2-tls-certs\" (UniqueName: \"kubernetes.io/secret/47691dfc-77d7-47dc-9922-d439e2de1fe2-tls-certs\") pod \"47691dfc-77d7-47dc-9922-d439e2de1fe2\" (UID: \"47691dfc-77d7-47dc-9922-d439e2de1fe2\") " Oct 30 22:53:44.592913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.592887 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-model-cache" pod "47691dfc-77d7-47dc-9922-d439e2de1fe2" (UID: "47691dfc-77d7-47dc-9922-d439e2de1fe2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:44.593121 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.593090 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-home" pod "47691dfc-77d7-47dc-9922-d439e2de1fe2" (UID: "47691dfc-77d7-47dc-9922-d439e2de1fe2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:44.593121 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.593096 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-tmp-dir" pod "47691dfc-77d7-47dc-9922-d439e2de1fe2" (UID: "47691dfc-77d7-47dc-9922-d439e2de1fe2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:44.593233 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.593214 4859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-home\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:44.593300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.593241 4859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-model-cache\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:44.593300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.593258 4859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-tmp-dir\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:44.594724 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.594696 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-dshm" pod "47691dfc-77d7-47dc-9922-d439e2de1fe2" (UID: "47691dfc-77d7-47dc-9922-d439e2de1fe2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:44.594813 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.594744 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47691dfc-77d7-47dc-9922-d439e2de1fe2-tls-certs" pod "47691dfc-77d7-47dc-9922-d439e2de1fe2" (UID: "47691dfc-77d7-47dc-9922-d439e2de1fe2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:44.594813 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.594785 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47691dfc-77d7-47dc-9922-d439e2de1fe2-kube-api-access-tdvr2" pod "47691dfc-77d7-47dc-9922-d439e2de1fe2" (UID: "47691dfc-77d7-47dc-9922-d439e2de1fe2"). InnerVolumeSpecName "kube-api-access-tdvr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:44.694343 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.694322 4859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47691dfc-77d7-47dc-9922-d439e2de1fe2-tls-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:44.694343 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.694342 4859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47691dfc-77d7-47dc-9922-d439e2de1fe2-dshm\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:44.694489 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:44.694352 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdvr2\" (UniqueName: \"kubernetes.io/projected/47691dfc-77d7-47dc-9922-d439e2de1fe2-kube-api-access-tdvr2\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.475365 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:45.475326 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" event={"ID":"47691dfc-77d7-47dc-9922-d439e2de1fe2","Type":"ContainerDied","Data":"e0401e5e2bc96c8a14eded44099ddd73a58b77a7ce288652a6a69c09e8a495ee"} Oct 30 22:53:45.475824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:45.475379 4859 scope.go:122] "RemoveContainer" containerID="aa055bf1a7b368f8021c261d1c3ad6926c51afb6c9ade4eb9345a3f5f165dda3" Oct 30 22:53:45.475824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:45.475375 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp" Oct 30 22:53:45.642234 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:45.642201 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp"] Oct 30 22:53:45.691175 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:45.691141 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/premium-w2-90e0d5-kserve-54bb6c8d8f-f9plp"] Oct 30 22:53:46.741167 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:46.741133 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" path="/var/lib/kubelet/pods/47691dfc-77d7-47dc-9922-d439e2de1fe2/volumes" Oct 30 22:53:53.391413 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:53.391375 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" podUID="f838b548-a378-4880-a793-bcdefc06f68b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:53.391878 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:53.391390 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" podUID="2f093449-0fc4-40b1-a47b-2dfdcd448e07" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:58.470320 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:53:58.470281 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:54:01.573580 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.573554 4859 generic.go:356] "Generic (PLEG): container finished" podID="f838b548-a378-4880-a793-bcdefc06f68b" containerID="8a166d3ff2c7af068e2bf7b2ec21c508cd82a0e29a7c9e3530f7e61cf79b7820" exitCode=0 Oct 30 22:54:01.573921 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.573623 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" event={"ID":"f838b548-a378-4880-a793-bcdefc06f68b","Type":"ContainerDied","Data":"8a166d3ff2c7af068e2bf7b2ec21c508cd82a0e29a7c9e3530f7e61cf79b7820"} Oct 30 22:54:01.718350 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.718280 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 22:54:01.826864 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.826836 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 22:54:01.876634 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.876609 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f838b548-a378-4880-a793-bcdefc06f68b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f838b548-a378-4880-a793-bcdefc06f68b-plugins-config-volume\") pod \"f838b548-a378-4880-a793-bcdefc06f68b\" (UID: \"f838b548-a378-4880-a793-bcdefc06f68b\") " Oct 30 22:54:01.876788 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.876654 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f838b548-a378-4880-a793-bcdefc06f68b-kube-api-access-xc6fr\" (UniqueName: \"kubernetes.io/projected/f838b548-a378-4880-a793-bcdefc06f68b-kube-api-access-xc6fr\") pod \"f838b548-a378-4880-a793-bcdefc06f68b\" (UID: \"f838b548-a378-4880-a793-bcdefc06f68b\") " Oct 30 22:54:01.876980 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.876933 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f838b548-a378-4880-a793-bcdefc06f68b-plugins-config-volume" pod "f838b548-a378-4880-a793-bcdefc06f68b" (UID: "f838b548-a378-4880-a793-bcdefc06f68b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:01.878685 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.878664 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f838b548-a378-4880-a793-bcdefc06f68b-kube-api-access-xc6fr" pod "f838b548-a378-4880-a793-bcdefc06f68b" (UID: "f838b548-a378-4880-a793-bcdefc06f68b"). InnerVolumeSpecName "kube-api-access-xc6fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:01.977448 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.977385 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f093449-0fc4-40b1-a47b-2dfdcd448e07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f093449-0fc4-40b1-a47b-2dfdcd448e07-plugins-config-volume\") pod \"2f093449-0fc4-40b1-a47b-2dfdcd448e07\" (UID: \"2f093449-0fc4-40b1-a47b-2dfdcd448e07\") " Oct 30 22:54:01.977448 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.977423 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f093449-0fc4-40b1-a47b-2dfdcd448e07-kube-api-access-vzwdl\" (UniqueName: \"kubernetes.io/projected/2f093449-0fc4-40b1-a47b-2dfdcd448e07-kube-api-access-vzwdl\") pod \"2f093449-0fc4-40b1-a47b-2dfdcd448e07\" (UID: \"2f093449-0fc4-40b1-a47b-2dfdcd448e07\") " Oct 30 22:54:01.977645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.977611 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f838b548-a378-4880-a793-bcdefc06f68b-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:01.977645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.977620 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6fr\" (UniqueName: \"kubernetes.io/projected/f838b548-a378-4880-a793-bcdefc06f68b-kube-api-access-xc6fr\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:01.977759 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.977695 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f093449-0fc4-40b1-a47b-2dfdcd448e07-plugins-config-volume" pod "2f093449-0fc4-40b1-a47b-2dfdcd448e07" (UID: "2f093449-0fc4-40b1-a47b-2dfdcd448e07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:01.979461 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:01.979440 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f093449-0fc4-40b1-a47b-2dfdcd448e07-kube-api-access-vzwdl" pod "2f093449-0fc4-40b1-a47b-2dfdcd448e07" (UID: "2f093449-0fc4-40b1-a47b-2dfdcd448e07"). InnerVolumeSpecName "kube-api-access-vzwdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:02.078283 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.078257 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f093449-0fc4-40b1-a47b-2dfdcd448e07-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:02.078283 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.078279 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwdl\" (UniqueName: \"kubernetes.io/projected/2f093449-0fc4-40b1-a47b-2dfdcd448e07-kube-api-access-vzwdl\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:02.580739 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.580706 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" Oct 30 22:54:02.581178 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.580737 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn" event={"ID":"f838b548-a378-4880-a793-bcdefc06f68b","Type":"ContainerDied","Data":"87af8fdadc2361027ea9e734bf2f2098d55e560bbd3afcdabbf830df36c2520b"} Oct 30 22:54:02.581178 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.580801 4859 scope.go:122] "RemoveContainer" containerID="8a166d3ff2c7af068e2bf7b2ec21c508cd82a0e29a7c9e3530f7e61cf79b7820" Oct 30 22:54:02.582188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.582057 4859 generic.go:356] "Generic (PLEG): container finished" podID="2f093449-0fc4-40b1-a47b-2dfdcd448e07" containerID="8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85" exitCode=0 Oct 30 22:54:02.582188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.582103 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" Oct 30 22:54:02.582188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.582118 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" event={"ID":"2f093449-0fc4-40b1-a47b-2dfdcd448e07","Type":"ContainerDied","Data":"8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85"} Oct 30 22:54:02.582188 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.582146 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq" event={"ID":"2f093449-0fc4-40b1-a47b-2dfdcd448e07","Type":"ContainerDied","Data":"0cc07f67cc2e76af857b8ac9b4f88cb52459b8dd074fe92b773f9c55b4897f2d"} Oct 30 22:54:02.591914 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.591788 4859 scope.go:122] "RemoveContainer" containerID="8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85" Oct 30 22:54:02.600132 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.600113 4859 scope.go:122] "RemoveContainer" containerID="8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85" Oct 30 22:54:02.600401 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:54:02.600370 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85\": container with ID starting with 8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85 not found: ID does not exist" containerID="8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85" Oct 30 22:54:02.600533 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.600409 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85"} err="failed to get container status \"8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85\": rpc error: code = NotFound desc = could not find container \"8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85\": container with ID starting with 8f232435fe5afba10dda98b4dbb68ede5f0f4821b5bde469b8264651e4369d85 not found: ID does not exist" Oct 30 22:54:02.743708 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.743650 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn"] Oct 30 22:54:02.793000 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.792980 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90e0d5-76ddd5dc65-vbhnn"] Oct 30 22:54:02.855710 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.855685 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq"] Oct 30 22:54:02.862960 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:02.862925 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90e0d5-79f87f5ff9-9jnkq"] Oct 30 22:54:04.739512 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:04.739483 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f093449-0fc4-40b1-a47b-2dfdcd448e07" path="/var/lib/kubelet/pods/2f093449-0fc4-40b1-a47b-2dfdcd448e07/volumes" Oct 30 22:54:04.739891 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:04.739846 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f838b548-a378-4880-a793-bcdefc06f68b" path="/var/lib/kubelet/pods/f838b548-a378-4880-a793-bcdefc06f68b/volumes" Oct 30 22:54:09.879631 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:54:09.879594 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:54:15.218913 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:15.218879 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:54:39.000680 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.000644 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp"] Oct 30 22:54:39.001083 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.000922 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" podUID="3aed1e80-3040-4b04-b64c-00c3c1db2da6" containerName="maas-api" containerID="cri-o://d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081" gracePeriod=30 Oct 30 22:54:39.263205 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.263148 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 22:54:39.355255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.355230 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aed1e80-3040-4b04-b64c-00c3c1db2da6-kube-api-access-qx2pj\" (UniqueName: \"kubernetes.io/projected/3aed1e80-3040-4b04-b64c-00c3c1db2da6-kube-api-access-qx2pj\") pod \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " Oct 30 22:54:39.355417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.355271 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-metrics-certs\") pod \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " Oct 30 22:54:39.355417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.355296 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-maas-api-tls\") pod \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " Oct 30 22:54:39.355417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.355320 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aed1e80-3040-4b04-b64c-00c3c1db2da6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aed1e80-3040-4b04-b64c-00c3c1db2da6-openshift-service-ca\") pod \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\" (UID: \"3aed1e80-3040-4b04-b64c-00c3c1db2da6\") " Oct 30 22:54:39.355736 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.355709 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aed1e80-3040-4b04-b64c-00c3c1db2da6-openshift-service-ca" pod "3aed1e80-3040-4b04-b64c-00c3c1db2da6" (UID: "3aed1e80-3040-4b04-b64c-00c3c1db2da6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:39.357551 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.357512 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aed1e80-3040-4b04-b64c-00c3c1db2da6-kube-api-access-qx2pj" pod "3aed1e80-3040-4b04-b64c-00c3c1db2da6" (UID: "3aed1e80-3040-4b04-b64c-00c3c1db2da6"). InnerVolumeSpecName "kube-api-access-qx2pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:39.357711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.357555 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-metrics-certs" pod "3aed1e80-3040-4b04-b64c-00c3c1db2da6" (UID: "3aed1e80-3040-4b04-b64c-00c3c1db2da6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:39.357711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.357700 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-maas-api-tls" pod "3aed1e80-3040-4b04-b64c-00c3c1db2da6" (UID: "3aed1e80-3040-4b04-b64c-00c3c1db2da6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:39.456184 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.456155 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx2pj\" (UniqueName: \"kubernetes.io/projected/3aed1e80-3040-4b04-b64c-00c3c1db2da6-kube-api-access-qx2pj\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:39.456184 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.456183 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:39.456357 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.456193 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aed1e80-3040-4b04-b64c-00c3c1db2da6-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:39.456357 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.456202 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aed1e80-3040-4b04-b64c-00c3c1db2da6-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:39.499475 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.499444 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:54:39.798525 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.798490 4859 generic.go:356] "Generic (PLEG): container finished" podID="3aed1e80-3040-4b04-b64c-00c3c1db2da6" containerID="d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081" exitCode=0 Oct 30 22:54:39.798687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.798537 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" Oct 30 22:54:39.798687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.798570 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" event={"ID":"3aed1e80-3040-4b04-b64c-00c3c1db2da6","Type":"ContainerDied","Data":"d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081"} Oct 30 22:54:39.798687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.798622 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp" event={"ID":"3aed1e80-3040-4b04-b64c-00c3c1db2da6","Type":"ContainerDied","Data":"dc3bb69073dc3886ac9a7067734eefd58304b9cf43710d7bdd9236120fbe5f1b"} Oct 30 22:54:39.798687 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.798646 4859 scope.go:122] "RemoveContainer" containerID="d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081" Oct 30 22:54:39.808546 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.808530 4859 scope.go:122] "RemoveContainer" containerID="d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081" Oct 30 22:54:39.808802 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:54:39.808785 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081\": container with ID starting with d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081 not found: ID does not exist" containerID="d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081" Oct 30 22:54:39.808857 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.808810 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081"} err="failed to get container status \"d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081\": rpc error: code = NotFound desc = could not find container \"d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081\": container with ID starting with d6388b59237b620c2b59ed6f4bc4975d348fb964589d2220f685a80923db2081 not found: ID does not exist" Oct 30 22:54:39.876677 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:54:39.876641 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:54:39.961803 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.961772 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp"] Oct 30 22:54:39.967389 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:39.967363 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90e0d5-764d49dc77-wbgmp"] Oct 30 22:54:40.740017 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:40.739980 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aed1e80-3040-4b04-b64c-00c3c1db2da6" path="/var/lib/kubelet/pods/3aed1e80-3040-4b04-b64c-00c3c1db2da6/volumes" Oct 30 22:54:48.187244 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:48.187210 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28"] Oct 30 22:54:48.187684 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:48.187401 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" podUID="ee846847-c342-4905-895f-d49a6199ffc2" containerName="istio-proxy" containerID="cri-o://89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a" gracePeriod=30 Oct 30 22:54:53.449094 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.449070 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:54:53.502067 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.501987 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee846847-c342-4905-895f-d49a6199ffc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee846847-c342-4905-895f-d49a6199ffc2-istiod-ca-cert\") pod \"ee846847-c342-4905-895f-d49a6199ffc2\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " Oct 30 22:54:53.502067 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502053 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-kube-api-access-xh7km\" (UniqueName: \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-kube-api-access-xh7km\") pod \"ee846847-c342-4905-895f-d49a6199ffc2\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " Oct 30 22:54:53.502273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502085 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-certs\") pod \"ee846847-c342-4905-895f-d49a6199ffc2\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " Oct 30 22:54:53.502273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502138 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-socket\") pod \"ee846847-c342-4905-895f-d49a6199ffc2\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " Oct 30 22:54:53.502273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502177 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-envoy\") pod \"ee846847-c342-4905-895f-d49a6199ffc2\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " Oct 30 22:54:53.502273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502216 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee846847-c342-4905-895f-d49a6199ffc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee846847-c342-4905-895f-d49a6199ffc2-istio-podinfo\") pod \"ee846847-c342-4905-895f-d49a6199ffc2\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " Oct 30 22:54:53.502480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502281 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-istio-token\" (UniqueName: \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-istio-token\") pod \"ee846847-c342-4905-895f-d49a6199ffc2\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " Oct 30 22:54:53.502480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502310 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-data\") pod \"ee846847-c342-4905-895f-d49a6199ffc2\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " Oct 30 22:54:53.502480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502404 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-credential-socket\") pod \"ee846847-c342-4905-895f-d49a6199ffc2\" (UID: \"ee846847-c342-4905-895f-d49a6199ffc2\") " Oct 30 22:54:53.502480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502441 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-certs" pod "ee846847-c342-4905-895f-d49a6199ffc2" (UID: "ee846847-c342-4905-895f-d49a6199ffc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:53.502480 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502451 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-socket" pod "ee846847-c342-4905-895f-d49a6199ffc2" (UID: "ee846847-c342-4905-895f-d49a6199ffc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:53.502711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502474 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee846847-c342-4905-895f-d49a6199ffc2-istiod-ca-cert" pod "ee846847-c342-4905-895f-d49a6199ffc2" (UID: "ee846847-c342-4905-895f-d49a6199ffc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:53.502711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502687 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-data" pod "ee846847-c342-4905-895f-d49a6199ffc2" (UID: "ee846847-c342-4905-895f-d49a6199ffc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:53.502819 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502736 4859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee846847-c342-4905-895f-d49a6199ffc2-istiod-ca-cert\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.502819 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502754 4859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.502819 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502768 4859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-workload-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.503019 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.502987 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-credential-socket" pod "ee846847-c342-4905-895f-d49a6199ffc2" (UID: "ee846847-c342-4905-895f-d49a6199ffc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:53.504643 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.504617 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-istio-token" pod "ee846847-c342-4905-895f-d49a6199ffc2" (UID: "ee846847-c342-4905-895f-d49a6199ffc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:53.504899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.504868 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee846847-c342-4905-895f-d49a6199ffc2-istio-podinfo" pod "ee846847-c342-4905-895f-d49a6199ffc2" (UID: "ee846847-c342-4905-895f-d49a6199ffc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:54:53.504899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.504885 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-kube-api-access-xh7km" pod "ee846847-c342-4905-895f-d49a6199ffc2" (UID: "ee846847-c342-4905-895f-d49a6199ffc2"). InnerVolumeSpecName "kube-api-access-xh7km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:53.504899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.504890 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-envoy" pod "ee846847-c342-4905-895f-d49a6199ffc2" (UID: "ee846847-c342-4905-895f-d49a6199ffc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:53.603905 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.603880 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7km\" (UniqueName: \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-kube-api-access-xh7km\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.603905 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.603905 4859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-envoy\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.604122 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.603914 4859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee846847-c342-4905-895f-d49a6199ffc2-istio-podinfo\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.604122 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.603925 4859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee846847-c342-4905-895f-d49a6199ffc2-istio-token\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.604122 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.603933 4859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-istio-data\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.604122 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.603954 4859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee846847-c342-4905-895f-d49a6199ffc2-credential-socket\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.879959 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.879912 4859 generic.go:356] "Generic (PLEG): container finished" podID="ee846847-c342-4905-895f-d49a6199ffc2" containerID="89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a" exitCode=0 Oct 30 22:54:53.880149 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.879966 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" event={"ID":"ee846847-c342-4905-895f-d49a6199ffc2","Type":"ContainerDied","Data":"89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a"} Oct 30 22:54:53.880149 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.880001 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" Oct 30 22:54:53.880149 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.880016 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28" event={"ID":"ee846847-c342-4905-895f-d49a6199ffc2","Type":"ContainerDied","Data":"f04ab3e4fd8da73aad1719a7b15fdaa4aef2574154ba6465ad499f5719c5ce99"} Oct 30 22:54:53.880149 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.880043 4859 scope.go:122] "RemoveContainer" containerID="89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a" Oct 30 22:54:53.891141 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.891118 4859 scope.go:122] "RemoveContainer" containerID="89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a" Oct 30 22:54:53.891415 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:54:53.891398 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a\": container with ID starting with 89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a not found: ID does not exist" containerID="89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a" Oct 30 22:54:53.891468 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:53.891423 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a"} err="failed to get container status \"89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a\": rpc error: code = NotFound desc = could not find container \"89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a\": container with ID starting with 89d3e6aa4ecff8815b8df1217523a48739fb2d5492a720e25b9f9035e5986b5a not found: ID does not exist" Oct 30 22:54:54.044073 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:54.044045 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28"] Oct 30 22:54:54.093509 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:54.093474 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-zxc28"] Oct 30 22:54:54.711897 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:54.711866 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:54:54.740728 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:54:54.740693 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee846847-c342-4905-895f-d49a6199ffc2" path="/var/lib/kubelet/pods/ee846847-c342-4905-895f-d49a6199ffc2/volumes" Oct 30 22:55:09.875278 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:55:09.875197 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:55:13.388134 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:55:13.388091 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:55:39.876844 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:55:39.876808 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:55:51.988049 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:55:51.988014 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:56:09.876633 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:56:09.876591 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:56:12.137417 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.137379 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs"] Oct 30 22:56:12.137905 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.137622 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" podUID="da81d643-efd0-4edb-a187-01a2e1a63641" containerName="maas-api" containerID="cri-o://6213dde8196dfd9e49d4a836c2100b46a1ad90e8d18ac6d3e8e904f2caf128a2" gracePeriod=30 Oct 30 22:56:12.328836 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.328806 4859 generic.go:356] "Generic (PLEG): container finished" podID="da81d643-efd0-4edb-a187-01a2e1a63641" containerID="6213dde8196dfd9e49d4a836c2100b46a1ad90e8d18ac6d3e8e904f2caf128a2" exitCode=0 Oct 30 22:56:12.329015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.328876 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" event={"ID":"da81d643-efd0-4edb-a187-01a2e1a63641","Type":"ContainerDied","Data":"6213dde8196dfd9e49d4a836c2100b46a1ad90e8d18ac6d3e8e904f2caf128a2"} Oct 30 22:56:12.396301 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.396240 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:56:12.512366 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.512331 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-maas-api-tls\") pod \"da81d643-efd0-4edb-a187-01a2e1a63641\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " Oct 30 22:56:12.512568 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.512391 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-metrics-certs\") pod \"da81d643-efd0-4edb-a187-01a2e1a63641\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " Oct 30 22:56:12.512568 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.512426 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da81d643-efd0-4edb-a187-01a2e1a63641-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da81d643-efd0-4edb-a187-01a2e1a63641-openshift-service-ca\") pod \"da81d643-efd0-4edb-a187-01a2e1a63641\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " Oct 30 22:56:12.512568 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.512477 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da81d643-efd0-4edb-a187-01a2e1a63641-kube-api-access-dcmsz\" (UniqueName: \"kubernetes.io/projected/da81d643-efd0-4edb-a187-01a2e1a63641-kube-api-access-dcmsz\") pod \"da81d643-efd0-4edb-a187-01a2e1a63641\" (UID: \"da81d643-efd0-4edb-a187-01a2e1a63641\") " Oct 30 22:56:12.512854 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.512818 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da81d643-efd0-4edb-a187-01a2e1a63641-openshift-service-ca" pod "da81d643-efd0-4edb-a187-01a2e1a63641" (UID: "da81d643-efd0-4edb-a187-01a2e1a63641"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:12.515263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.515089 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-maas-api-tls" pod "da81d643-efd0-4edb-a187-01a2e1a63641" (UID: "da81d643-efd0-4edb-a187-01a2e1a63641"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:12.515263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.515206 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-metrics-certs" pod "da81d643-efd0-4edb-a187-01a2e1a63641" (UID: "da81d643-efd0-4edb-a187-01a2e1a63641"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:12.515263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.515235 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da81d643-efd0-4edb-a187-01a2e1a63641-kube-api-access-dcmsz" pod "da81d643-efd0-4edb-a187-01a2e1a63641" (UID: "da81d643-efd0-4edb-a187-01a2e1a63641"). InnerVolumeSpecName "kube-api-access-dcmsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:12.613408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.613375 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-maas-api-tls\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:12.613408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.613404 4859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da81d643-efd0-4edb-a187-01a2e1a63641-metrics-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:12.613408 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.613414 4859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da81d643-efd0-4edb-a187-01a2e1a63641-openshift-service-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:12.613650 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:12.613423 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcmsz\" (UniqueName: \"kubernetes.io/projected/da81d643-efd0-4edb-a187-01a2e1a63641-kube-api-access-dcmsz\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:13.335428 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:13.335399 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" Oct 30 22:56:13.335983 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:13.335439 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs" event={"ID":"da81d643-efd0-4edb-a187-01a2e1a63641","Type":"ContainerDied","Data":"c8a48e260c0fbc92aae093a40b5a03a659026e5685014ccbcaba4ae4ba5fc1a0"} Oct 30 22:56:13.335983 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:13.335509 4859 scope.go:122] "RemoveContainer" containerID="6213dde8196dfd9e49d4a836c2100b46a1ad90e8d18ac6d3e8e904f2caf128a2" Oct 30 22:56:13.499015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:13.498974 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs"] Oct 30 22:56:13.548490 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:13.548454 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-89cf28-d6884457d-5c2gs"] Oct 30 22:56:14.739433 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:14.739399 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da81d643-efd0-4edb-a187-01a2e1a63641" path="/var/lib/kubelet/pods/da81d643-efd0-4edb-a187-01a2e1a63641/volumes" Oct 30 22:56:17.863463 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:17.863420 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:56:25.044933 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.044895 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd"] Oct 30 22:56:25.045361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045316 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3db3fd91-9283-4244-966d-4ac6378ce512" containerName="payload-processing" Oct 30 22:56:25.045361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045328 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db3fd91-9283-4244-966d-4ac6378ce512" containerName="payload-processing" Oct 30 22:56:25.045361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045338 4859 cpu_manager.go:414] "containerMap: removing container" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" Oct 30 22:56:25.045361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045343 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" Oct 30 22:56:25.045361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045350 4859 cpu_manager.go:414] "containerMap: removing container" podUID="4f8242e9-9b3b-429a-ad38-09dad5c6ef30" containerName="maas-api" Oct 30 22:56:25.045361 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045355 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8242e9-9b3b-429a-ad38-09dad5c6ef30" containerName="maas-api" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045366 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045371 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045379 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045384 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045394 4859 cpu_manager.go:414] "containerMap: removing container" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045400 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045410 4859 cpu_manager.go:414] "containerMap: removing container" podUID="1f3d68b6-bfaa-4994-9ace-6eedae9df25f" containerName="payload-processing" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045414 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3d68b6-bfaa-4994-9ace-6eedae9df25f" containerName="payload-processing" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045421 4859 cpu_manager.go:414] "containerMap: removing container" podUID="ec42687d-7e39-4655-8873-8f790f18baf6" containerName="main" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045425 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec42687d-7e39-4655-8873-8f790f18baf6" containerName="main" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045433 4859 cpu_manager.go:414] "containerMap: removing container" podUID="da81d643-efd0-4edb-a187-01a2e1a63641" containerName="maas-api" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045437 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da81d643-efd0-4edb-a187-01a2e1a63641" containerName="maas-api" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045443 4859 cpu_manager.go:414] "containerMap: removing container" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045447 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045454 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045459 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045464 4859 cpu_manager.go:414] "containerMap: removing container" podUID="082c36fb-0359-48b9-bd96-7248ee570108" containerName="maas-api" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045468 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082c36fb-0359-48b9-bd96-7248ee570108" containerName="maas-api" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045475 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e68cb285-6e5e-4918-9962-fd8cc8e8680f" containerName="maas-api" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045479 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68cb285-6e5e-4918-9962-fd8cc8e8680f" containerName="maas-api" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045486 4859 cpu_manager.go:414] "containerMap: removing container" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045490 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045496 4859 cpu_manager.go:414] "containerMap: removing container" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045500 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045508 4859 cpu_manager.go:414] "containerMap: removing container" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045513 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045524 4859 cpu_manager.go:414] "containerMap: removing container" podUID="ee846847-c342-4905-895f-d49a6199ffc2" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045528 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee846847-c342-4905-895f-d49a6199ffc2" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045535 4859 cpu_manager.go:414] "containerMap: removing container" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045539 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045546 4859 cpu_manager.go:414] "containerMap: removing container" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045551 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045559 4859 cpu_manager.go:414] "containerMap: removing container" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045564 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" Oct 30 22:56:25.045553 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045572 4859 cpu_manager.go:414] "containerMap: removing container" podUID="59fc40d6-745b-42b8-9918-cb6ae52ff6aa" containerName="payload-pre-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045577 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59fc40d6-745b-42b8-9918-cb6ae52ff6aa" containerName="payload-pre-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045584 4859 cpu_manager.go:414] "containerMap: removing container" podUID="2f093449-0fc4-40b1-a47b-2dfdcd448e07" containerName="payload-pre-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045588 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f093449-0fc4-40b1-a47b-2dfdcd448e07" containerName="payload-pre-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045597 4859 cpu_manager.go:414] "containerMap: removing container" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045601 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045609 4859 cpu_manager.go:414] "containerMap: removing container" podUID="998baa49-be51-4cd6-95d3-b7e6288f974e" containerName="payload-pre-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045614 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998baa49-be51-4cd6-95d3-b7e6288f974e" containerName="payload-pre-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045620 4859 cpu_manager.go:414] "containerMap: removing container" podUID="336cf7be-cb8c-4938-b19e-e37f6f99b165" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045624 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336cf7be-cb8c-4938-b19e-e37f6f99b165" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045631 4859 cpu_manager.go:414] "containerMap: removing container" podUID="964b48c3-e385-4359-9b3f-91df3e092b08" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045635 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964b48c3-e385-4359-9b3f-91df3e092b08" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045641 4859 cpu_manager.go:414] "containerMap: removing container" podUID="0ec2eb4a-892d-4311-b638-4e2330270d73" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045645 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec2eb4a-892d-4311-b638-4e2330270d73" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045651 4859 cpu_manager.go:414] "containerMap: removing container" podUID="f838b548-a378-4880-a793-bcdefc06f68b" containerName="payload-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045656 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f838b548-a378-4880-a793-bcdefc06f68b" containerName="payload-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045662 4859 cpu_manager.go:414] "containerMap: removing container" podUID="3aed1e80-3040-4b04-b64c-00c3c1db2da6" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045666 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aed1e80-3040-4b04-b64c-00c3c1db2da6" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045671 4859 cpu_manager.go:414] "containerMap: removing container" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045675 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045733 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3d68b6-bfaa-4994-9ace-6eedae9df25f" containerName="payload-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045743 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec42687d-7e39-4655-8873-8f790f18baf6" containerName="main" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045750 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="da81d643-efd0-4edb-a187-01a2e1a63641" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045757 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045761 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045765 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="082c36fb-0359-48b9-bd96-7248ee570108" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045772 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68cb285-6e5e-4918-9962-fd8cc8e8680f" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045777 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="47691dfc-77d7-47dc-9922-d439e2de1fe2" containerName="main" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045783 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045789 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1491e9-ac51-4e0b-828a-fe8984ff6282" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045794 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee846847-c342-4905-895f-d49a6199ffc2" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045800 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb061db-828f-4cc8-aa1f-116685012e01" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045805 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045809 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d35c973-cc54-443b-b0a4-add6d36a409c" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045814 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="59fc40d6-745b-42b8-9918-cb6ae52ff6aa" containerName="payload-pre-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045819 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f093449-0fc4-40b1-a47b-2dfdcd448e07" containerName="payload-pre-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045824 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045829 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="998baa49-be51-4cd6-95d3-b7e6288f974e" containerName="payload-pre-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045834 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="336cf7be-cb8c-4938-b19e-e37f6f99b165" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045839 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="964b48c3-e385-4359-9b3f-91df3e092b08" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045844 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec2eb4a-892d-4311-b638-4e2330270d73" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045849 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045852 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f838b548-a378-4880-a793-bcdefc06f68b" containerName="payload-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045857 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aed1e80-3040-4b04-b64c-00c3c1db2da6" containerName="maas-api" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045861 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc053df-ca6b-49b5-ab28-04b0fafa67e2" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045866 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db3fd91-9283-4244-966d-4ac6378ce512" containerName="payload-processing" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045872 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" Oct 30 22:56:25.046653 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.045876 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8242e9-9b3b-429a-ad38-09dad5c6ef30" containerName="maas-api" Oct 30 22:56:25.050399 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.050382 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:25.106047 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.106020 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd"] Oct 30 22:56:25.152450 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.152422 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:56:25.152684 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.152665 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:56:25.152732 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.152700 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:56:25.152732 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.152721 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\"" Oct 30 22:56:25.236834 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.236809 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:25.236987 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.236875 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z94c6\" (UniqueName: \"kubernetes.io/projected/403a7402-c8af-4324-8e60-d21850f9d68c-kube-api-access-z94c6\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:25.236987 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.236973 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:25.337559 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.337536 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z94c6\" (UniqueName: \"kubernetes.io/projected/403a7402-c8af-4324-8e60-d21850f9d68c-kube-api-access-z94c6\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:25.337704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.337575 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:25.337704 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.337671 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:25.338504 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.338480 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:25.340018 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.339996 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:25.395760 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.395733 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z94c6\" (UniqueName: \"kubernetes.io/projected/403a7402-c8af-4324-8e60-d21850f9d68c-kube-api-access-z94c6\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:25.662915 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:25.662843 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:26.047032 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:26.047005 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd"] Oct 30 22:56:26.049171 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:56:26.049145 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403a7402_c8af_4324_8e60_d21850f9d68c.slice/crio-3384b286a77bbcc710264c8ea155dbe65c0bacb15b064e0dfc57a30c3b6181a5 WatchSource:0}: Error finding container 3384b286a77bbcc710264c8ea155dbe65c0bacb15b064e0dfc57a30c3b6181a5: Status 404 returned error can't find the container with id 3384b286a77bbcc710264c8ea155dbe65c0bacb15b064e0dfc57a30c3b6181a5 Oct 30 22:56:26.418429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:26.418401 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" event={"ID":"403a7402-c8af-4324-8e60-d21850f9d68c","Type":"ContainerStarted","Data":"4b757ad5255238017d2e08bb28565426eb97ad043beb87eea94eb272cffb7607"} Oct 30 22:56:26.418533 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:26.418450 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" event={"ID":"403a7402-c8af-4324-8e60-d21850f9d68c","Type":"ContainerStarted","Data":"3384b286a77bbcc710264c8ea155dbe65c0bacb15b064e0dfc57a30c3b6181a5"} Oct 30 22:56:26.533500 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:26.533455 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" podStartSLOduration=1.533441177 podStartE2EDuration="1.533441177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:26.531817588 +0000 UTC m=+4443.904444334" watchObservedRunningTime="2026-10-30 22:56:26.533441177 +0000 UTC m=+4443.906067922" Oct 30 22:56:27.425213 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:27.425178 4859 generic.go:356] "Generic (PLEG): container finished" podID="403a7402-c8af-4324-8e60-d21850f9d68c" containerID="4b757ad5255238017d2e08bb28565426eb97ad043beb87eea94eb272cffb7607" exitCode=0 Oct 30 22:56:27.425620 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:27.425257 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" event={"ID":"403a7402-c8af-4324-8e60-d21850f9d68c","Type":"ContainerDied","Data":"4b757ad5255238017d2e08bb28565426eb97ad043beb87eea94eb272cffb7607"} Oct 30 22:56:28.566010 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:28.565987 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:28.667764 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:28.667735 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-xks-ca\") pod \"403a7402-c8af-4324-8e60-d21850f9d68c\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " Oct 30 22:56:28.667892 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:28.667801 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403a7402-c8af-4324-8e60-d21850f9d68c-kube-api-access-z94c6\" (UniqueName: \"kubernetes.io/projected/403a7402-c8af-4324-8e60-d21850f9d68c-kube-api-access-z94c6\") pod \"403a7402-c8af-4324-8e60-d21850f9d68c\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " Oct 30 22:56:28.667892 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:28.667851 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-openshift-ca\") pod \"403a7402-c8af-4324-8e60-d21850f9d68c\" (UID: \"403a7402-c8af-4324-8e60-d21850f9d68c\") " Oct 30 22:56:28.668264 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:28.668238 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-openshift-ca" pod "403a7402-c8af-4324-8e60-d21850f9d68c" (UID: "403a7402-c8af-4324-8e60-d21850f9d68c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:28.669834 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:28.669810 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-xks-ca" pod "403a7402-c8af-4324-8e60-d21850f9d68c" (UID: "403a7402-c8af-4324-8e60-d21850f9d68c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:28.669919 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:28.669837 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403a7402-c8af-4324-8e60-d21850f9d68c-kube-api-access-z94c6" pod "403a7402-c8af-4324-8e60-d21850f9d68c" (UID: "403a7402-c8af-4324-8e60-d21850f9d68c"). InnerVolumeSpecName "kube-api-access-z94c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:28.768821 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:28.768796 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-xks-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:28.768821 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:28.768822 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z94c6\" (UniqueName: \"kubernetes.io/projected/403a7402-c8af-4324-8e60-d21850f9d68c-kube-api-access-z94c6\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:28.768982 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:28.768832 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403a7402-c8af-4324-8e60-d21850f9d68c-maas-api-openshift-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:29.441026 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:29.440989 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" Oct 30 22:56:29.441200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:29.440986 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd" event={"ID":"403a7402-c8af-4324-8e60-d21850f9d68c","Type":"ContainerDied","Data":"3384b286a77bbcc710264c8ea155dbe65c0bacb15b064e0dfc57a30c3b6181a5"} Oct 30 22:56:29.441200 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:29.441129 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3384b286a77bbcc710264c8ea155dbe65c0bacb15b064e0dfc57a30c3b6181a5" Oct 30 22:56:39.876361 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:56:39.876286 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:56:45.615824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:45.615788 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd"] Oct 30 22:56:45.665391 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:45.665361 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e450a-r6kbd"] Oct 30 22:56:46.740232 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:46.740197 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403a7402-c8af-4324-8e60-d21850f9d68c" path="/var/lib/kubelet/pods/403a7402-c8af-4324-8e60-d21850f9d68c/volumes" Oct 30 22:56:48.907035 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:48.906999 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:56:58.330442 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:56:58.330406 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj"] Oct 30 22:57:05.330779 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:05.330744 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt"] Oct 30 22:57:09.875770 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:57:09.875733 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:57:09.939476 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:09.939444 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5"] Oct 30 22:57:09.939862 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:09.939850 4859 cpu_manager.go:414] "containerMap: removing container" podUID="403a7402-c8af-4324-8e60-d21850f9d68c" containerName="revoke-keys" Oct 30 22:57:09.939907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:09.939864 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403a7402-c8af-4324-8e60-d21850f9d68c" containerName="revoke-keys" Oct 30 22:57:09.939907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:09.939889 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" Oct 30 22:57:09.939907 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:09.939894 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" Oct 30 22:57:09.940023 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:09.939981 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="403a7402-c8af-4324-8e60-d21850f9d68c" containerName="revoke-keys" Oct 30 22:57:09.940023 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:09.939999 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3efccd2-b4f7-47ad-ac24-326f5246aac6" containerName="istio-proxy" Oct 30 22:57:09.940023 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:09.940009 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="825d33d5-6db2-4c6b-a8a9-bd5a8a615449" containerName="istio-proxy" Oct 30 22:57:09.940023 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:09.940018 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f42837-1b1c-4f5b-9236-16ed9c6cb95b" containerName="istio-proxy" Oct 30 22:57:09.943034 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:09.943019 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.001815 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.001788 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5"] Oct 30 22:57:10.044407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.044376 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:57:10.044407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.044389 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\"" Oct 30 22:57:10.044407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.044389 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:57:10.044606 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.044422 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:57:10.133240 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.133174 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cdf242-pchf5\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.133240 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.133208 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cdf242-pchf5\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.133423 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.133315 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6znc\" (UniqueName: \"kubernetes.io/projected/e6a27b5c-b64d-40a6-be10-12011e09b3ee-kube-api-access-l6znc\") pod \"maas-api-revoke-sub-d9cdf242-pchf5\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.234739 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.234704 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cdf242-pchf5\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.234911 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.234746 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cdf242-pchf5\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.234911 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.234822 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6znc\" (UniqueName: \"kubernetes.io/projected/e6a27b5c-b64d-40a6-be10-12011e09b3ee-kube-api-access-l6znc\") pod \"maas-api-revoke-sub-d9cdf242-pchf5\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.235423 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.235401 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cdf242-pchf5\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.236954 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.236929 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cdf242-pchf5\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.291424 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.291396 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6znc\" (UniqueName: \"kubernetes.io/projected/e6a27b5c-b64d-40a6-be10-12011e09b3ee-kube-api-access-l6znc\") pod \"maas-api-revoke-sub-d9cdf242-pchf5\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.331283 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.331252 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" podUID="39d7d334-d9ba-4857-9161-3c5fc602194b" containerName="payload-processing" containerID="cri-o://0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842" gracePeriod=25 Oct 30 22:57:10.556116 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.556038 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:10.926497 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:10.926471 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5"] Oct 30 22:57:10.928389 ip-10-0-141-104 kubenswrapper[4859]: W1030 22:57:10.928356 4859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a27b5c_b64d_40a6_be10_12011e09b3ee.slice/crio-98acbecb9ed08fd603b3195a2430707753998da122e698309b2799af9f355602 WatchSource:0}: Error finding container 98acbecb9ed08fd603b3195a2430707753998da122e698309b2799af9f355602: Status 404 returned error can't find the container with id 98acbecb9ed08fd603b3195a2430707753998da122e698309b2799af9f355602 Oct 30 22:57:11.682524 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:11.682486 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" event={"ID":"e6a27b5c-b64d-40a6-be10-12011e09b3ee","Type":"ContainerStarted","Data":"fb2e8b3195b20d9af0dd46b79bbc650aaba6e7ce6b3ae6c171856ff27f7a8325"} Oct 30 22:57:11.682699 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:11.682541 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" event={"ID":"e6a27b5c-b64d-40a6-be10-12011e09b3ee","Type":"ContainerStarted","Data":"98acbecb9ed08fd603b3195a2430707753998da122e698309b2799af9f355602"} Oct 30 22:57:11.797719 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:11.797650 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" podStartSLOduration=2.7976335690000003 podStartE2EDuration="2.797633569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:11.796370784 +0000 UTC m=+4489.168997530" watchObservedRunningTime="2026-10-30 22:57:11.797633569 +0000 UTC m=+4489.170260316" Oct 30 22:57:12.689194 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:12.689156 4859 generic.go:356] "Generic (PLEG): container finished" podID="e6a27b5c-b64d-40a6-be10-12011e09b3ee" containerID="fb2e8b3195b20d9af0dd46b79bbc650aaba6e7ce6b3ae6c171856ff27f7a8325" exitCode=0 Oct 30 22:57:12.689585 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:12.689221 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" event={"ID":"e6a27b5c-b64d-40a6-be10-12011e09b3ee","Type":"ContainerDied","Data":"fb2e8b3195b20d9af0dd46b79bbc650aaba6e7ce6b3ae6c171856ff27f7a8325"} Oct 30 22:57:13.349985 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.349916 4859 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" containerID="cri-o://51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b" gracePeriod=45 Oct 30 22:57:13.354364 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.354330 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": EOF" Oct 30 22:57:13.624103 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.624076 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:57:13.693158 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.693118 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-dshm\") pod \"a5de8966-9960-46fb-8704-ae704cc77a6e\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " Oct 30 22:57:13.693635 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.693169 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-home\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-home\") pod \"a5de8966-9960-46fb-8704-ae704cc77a6e\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " Oct 30 22:57:13.693635 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.693199 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-tmp-dir\") pod \"a5de8966-9960-46fb-8704-ae704cc77a6e\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " Oct 30 22:57:13.693635 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.693252 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5de8966-9960-46fb-8704-ae704cc77a6e-kube-api-access-qj4pg\" (UniqueName: \"kubernetes.io/projected/a5de8966-9960-46fb-8704-ae704cc77a6e-kube-api-access-qj4pg\") pod \"a5de8966-9960-46fb-8704-ae704cc77a6e\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " Oct 30 22:57:13.693635 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.693280 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-model-cache\") pod \"a5de8966-9960-46fb-8704-ae704cc77a6e\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " Oct 30 22:57:13.693635 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.693372 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5de8966-9960-46fb-8704-ae704cc77a6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5de8966-9960-46fb-8704-ae704cc77a6e-tls-certs\") pod \"a5de8966-9960-46fb-8704-ae704cc77a6e\" (UID: \"a5de8966-9960-46fb-8704-ae704cc77a6e\") " Oct 30 22:57:13.693635 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.693599 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-model-cache" pod "a5de8966-9960-46fb-8704-ae704cc77a6e" (UID: "a5de8966-9960-46fb-8704-ae704cc77a6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:13.693973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.693702 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-home" pod "a5de8966-9960-46fb-8704-ae704cc77a6e" (UID: "a5de8966-9960-46fb-8704-ae704cc77a6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:13.693973 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.693732 4859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-model-cache\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:13.694340 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.694305 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-tmp-dir" pod "a5de8966-9960-46fb-8704-ae704cc77a6e" (UID: "a5de8966-9960-46fb-8704-ae704cc77a6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:13.696324 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.696263 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5de8966-9960-46fb-8704-ae704cc77a6e-tls-certs" pod "a5de8966-9960-46fb-8704-ae704cc77a6e" (UID: "a5de8966-9960-46fb-8704-ae704cc77a6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:13.698460 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.698430 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5de8966-9960-46fb-8704-ae704cc77a6e-kube-api-access-qj4pg" pod "a5de8966-9960-46fb-8704-ae704cc77a6e" (UID: "a5de8966-9960-46fb-8704-ae704cc77a6e"). InnerVolumeSpecName "kube-api-access-qj4pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:13.698818 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.698780 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-dshm" pod "a5de8966-9960-46fb-8704-ae704cc77a6e" (UID: "a5de8966-9960-46fb-8704-ae704cc77a6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:13.699187 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.699152 4859 generic.go:356] "Generic (PLEG): container finished" podID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerID="51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b" exitCode=0 Oct 30 22:57:13.699497 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.699464 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" event={"ID":"a5de8966-9960-46fb-8704-ae704cc77a6e","Type":"ContainerDied","Data":"51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b"} Oct 30 22:57:13.699605 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.699472 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" Oct 30 22:57:13.699605 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.699536 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj" event={"ID":"a5de8966-9960-46fb-8704-ae704cc77a6e","Type":"ContainerDied","Data":"21fe06f70c69b6c0a37e66c15770c47ce3be9c37ed57efab77d98b2ee67d5a64"} Oct 30 22:57:13.699605 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.699565 4859 scope.go:122] "RemoveContainer" containerID="51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b" Oct 30 22:57:13.737133 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.731257 4859 scope.go:122] "RemoveContainer" containerID="51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b" Oct 30 22:57:13.737133 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:57:13.731650 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b\": container with ID starting with 51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b not found: ID does not exist" containerID="51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b" Oct 30 22:57:13.737133 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.731693 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b"} err="failed to get container status \"51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b\": rpc error: code = NotFound desc = could not find container \"51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b\": container with ID starting with 51c71943cde0b27237665b6c1177b4be9f1dfd0baf4a4afd182549215d864a1b not found: ID does not exist" Oct 30 22:57:13.798986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.797585 4859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-dshm\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:13.798986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.797615 4859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-home\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:13.798986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.797628 4859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5de8966-9960-46fb-8704-ae704cc77a6e-tmp-dir\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:13.798986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.797641 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj4pg\" (UniqueName: \"kubernetes.io/projected/a5de8966-9960-46fb-8704-ae704cc77a6e-kube-api-access-qj4pg\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:13.798986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.797655 4859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5de8966-9960-46fb-8704-ae704cc77a6e-tls-certs\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:13.868015 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.867008 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj"] Oct 30 22:57:13.871008 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.869068 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-w3-e90e35-kserve-855889c455-dhnsj"] Oct 30 22:57:13.885989 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.885251 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:14.000964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.998639 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-xks-ca\") pod \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " Oct 30 22:57:14.000964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.998705 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-openshift-ca\") pod \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " Oct 30 22:57:14.000964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.998868 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a27b5c-b64d-40a6-be10-12011e09b3ee-kube-api-access-l6znc\" (UniqueName: \"kubernetes.io/projected/e6a27b5c-b64d-40a6-be10-12011e09b3ee-kube-api-access-l6znc\") pod \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\" (UID: \"e6a27b5c-b64d-40a6-be10-12011e09b3ee\") " Oct 30 22:57:14.000964 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:13.999639 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-openshift-ca" pod "e6a27b5c-b64d-40a6-be10-12011e09b3ee" (UID: "e6a27b5c-b64d-40a6-be10-12011e09b3ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:14.002395 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:14.002364 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a27b5c-b64d-40a6-be10-12011e09b3ee-kube-api-access-l6znc" pod "e6a27b5c-b64d-40a6-be10-12011e09b3ee" (UID: "e6a27b5c-b64d-40a6-be10-12011e09b3ee"). InnerVolumeSpecName "kube-api-access-l6znc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:14.005120 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:14.004906 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-xks-ca" pod "e6a27b5c-b64d-40a6-be10-12011e09b3ee" (UID: "e6a27b5c-b64d-40a6-be10-12011e09b3ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:14.100770 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:14.100737 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-xks-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:14.100770 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:14.100769 4859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a27b5c-b64d-40a6-be10-12011e09b3ee-maas-api-openshift-ca\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:14.100986 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:14.100781 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6znc\" (UniqueName: \"kubernetes.io/projected/e6a27b5c-b64d-40a6-be10-12011e09b3ee-kube-api-access-l6znc\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:14.712075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:14.711982 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" event={"ID":"e6a27b5c-b64d-40a6-be10-12011e09b3ee","Type":"ContainerDied","Data":"98acbecb9ed08fd603b3195a2430707753998da122e698309b2799af9f355602"} Oct 30 22:57:14.712075 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:14.712012 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cdf242-pchf5" Oct 30 22:57:14.712557 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:14.712083 4859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98acbecb9ed08fd603b3195a2430707753998da122e698309b2799af9f355602" Oct 30 22:57:14.740368 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:14.740340 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" path="/var/lib/kubelet/pods/a5de8966-9960-46fb-8704-ae704cc77a6e/volumes" Oct 30 22:57:17.193935 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:17.193889 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" podUID="39d7d334-d9ba-4857-9161-3c5fc602194b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:57:23.566548 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:23.566515 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:57:23.575197 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:23.575170 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:57:23.593401 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:23.593375 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c485nw_e7997836-978a-4993-af32-0631995a3099/extract/0.log" Oct 30 22:57:23.601429 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:23.601406 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthdr_f9d15d9a-71ac-4572-98ca-c5f5df251cb2/console-operator/2.log" Oct 30 22:57:26.009571 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:26.009536 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:57:27.193793 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:27.193756 4859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" podUID="39d7d334-d9ba-4857-9161-3c5fc602194b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:57:30.587263 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.587241 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 22:57:30.654446 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.654420 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d7d334-d9ba-4857-9161-3c5fc602194b-kube-api-access-wfl42\" (UniqueName: \"kubernetes.io/projected/39d7d334-d9ba-4857-9161-3c5fc602194b-kube-api-access-wfl42\") pod \"39d7d334-d9ba-4857-9161-3c5fc602194b\" (UID: \"39d7d334-d9ba-4857-9161-3c5fc602194b\") " Oct 30 22:57:30.654597 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.654529 4859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d7d334-d9ba-4857-9161-3c5fc602194b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d7d334-d9ba-4857-9161-3c5fc602194b-plugins-config-volume\") pod \"39d7d334-d9ba-4857-9161-3c5fc602194b\" (UID: \"39d7d334-d9ba-4857-9161-3c5fc602194b\") " Oct 30 22:57:30.654910 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.654882 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d7d334-d9ba-4857-9161-3c5fc602194b-plugins-config-volume" pod "39d7d334-d9ba-4857-9161-3c5fc602194b" (UID: "39d7d334-d9ba-4857-9161-3c5fc602194b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:30.656492 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.656472 4859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d7d334-d9ba-4857-9161-3c5fc602194b-kube-api-access-wfl42" pod "39d7d334-d9ba-4857-9161-3c5fc602194b" (UID: "39d7d334-d9ba-4857-9161-3c5fc602194b"). InnerVolumeSpecName "kube-api-access-wfl42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:30.756108 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.756081 4859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d7d334-d9ba-4857-9161-3c5fc602194b-plugins-config-volume\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:30.756108 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.756106 4859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfl42\" (UniqueName: \"kubernetes.io/projected/39d7d334-d9ba-4857-9161-3c5fc602194b-kube-api-access-wfl42\") on node \"ip-10-0-141-104.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:30.804596 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.804560 4859 generic.go:356] "Generic (PLEG): container finished" podID="39d7d334-d9ba-4857-9161-3c5fc602194b" containerID="0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842" exitCode=0 Oct 30 22:57:30.804755 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.804602 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" event={"ID":"39d7d334-d9ba-4857-9161-3c5fc602194b","Type":"ContainerDied","Data":"0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842"} Oct 30 22:57:30.804755 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.804619 4859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" Oct 30 22:57:30.804755 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.804669 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt" event={"ID":"39d7d334-d9ba-4857-9161-3c5fc602194b","Type":"ContainerDied","Data":"09f05e0d2c776ed4ce27fb006ab062bd43cd29640f349e611011b605fa2ece2a"} Oct 30 22:57:30.804755 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.804704 4859 scope.go:122] "RemoveContainer" containerID="0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842" Oct 30 22:57:30.814376 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.814333 4859 scope.go:122] "RemoveContainer" containerID="0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842" Oct 30 22:57:30.814713 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:57:30.814688 4859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842\": container with ID starting with 0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842 not found: ID does not exist" containerID="0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842" Oct 30 22:57:30.814824 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.814725 4859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842"} err="failed to get container status \"0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842\": rpc error: code = NotFound desc = could not find container \"0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842\": container with ID starting with 0933ba749709ef050bba60d92c4cfbcb0b6455185cdfea2c0938e07a99de3842 not found: ID does not exist" Oct 30 22:57:30.970035 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.969963 4859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt"] Oct 30 22:57:30.970035 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:30.970004 4859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e90e35-8647584c55-b2zbt"] Oct 30 22:57:32.740133 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:32.740100 4859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d7d334-d9ba-4857-9161-3c5fc602194b" path="/var/lib/kubelet/pods/39d7d334-d9ba-4857-9161-3c5fc602194b/volumes" Oct 30 22:57:35.979298 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:35.979267 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77dd9fff5-h8rlb_2d74b772-c978-43f7-a84b-f23f68145de2/authorino/0.log" Oct 30 22:57:36.388273 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:36.388246 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-llb2k_019fe412-f0e3-48d4-9878-963b1a067f08/manager/0.log" Oct 30 22:57:36.627673 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:36.627645 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tmw28_067cbc96-3d21-422f-b69f-ffa11e3c5706/manager/0.log" Oct 30 22:57:38.009795 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:38.009765 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-l56rn_512e3b58-f29f-404d-8fef-4675058fac54/manager/0.log" Oct 30 22:57:38.225998 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:38.225968 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4xv26_0591cd90-06f6-40c3-a2bf-411dad660142/limitador/0.log" Oct 30 22:57:38.444492 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:38.444470 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z6cfl_a300142a-c951-45c6-832a-ec9949b961d5/manager/0.log" Oct 30 22:57:39.374426 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:39.374394 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4xv26"] Oct 30 22:57:39.712351 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:39.712279 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lwspj_dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1/manager/0.log" Oct 30 22:57:39.878851 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:57:39.878815 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:57:41.059374 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:41.059349 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lv5fm_36da7174-ce14-4f51-ae5e-21bd5b72fcae/server/0.log" Oct 30 22:57:41.554953 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:41.554925 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qf8tr_a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda/proxy/0.log" Oct 30 22:57:42.289073 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:57:42.289048 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77dd9fff5-h8rlb_2d74b772-c978-43f7-a84b-f23f68145de2/authorino/0.log" Oct 30 22:58:09.881586 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:58:09.881500 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:58:12.500072 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:12.500042 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lwspj_dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1/manager/0.log" Oct 30 22:58:14.063630 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:14.063602 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lv5fm_36da7174-ce14-4f51-ae5e-21bd5b72fcae/server/0.log" Oct 30 22:58:14.606136 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:14.606105 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qf8tr_a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda/proxy/0.log" Oct 30 22:58:16.260850 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:16.260820 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d9cdf242-pchf5_e6a27b5c-b64d-40a6-be10-12011e09b3ee/revoke-keys/0.log" Oct 30 22:58:18.289875 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:18.289837 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77dd9fff5-h8rlb_2d74b772-c978-43f7-a84b-f23f68145de2/authorino/0.log" Oct 30 22:58:18.656747 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:18.656715 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-llb2k_019fe412-f0e3-48d4-9878-963b1a067f08/manager/0.log" Oct 30 22:58:18.887503 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:18.887478 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tmw28_067cbc96-3d21-422f-b69f-ffa11e3c5706/manager/0.log" Oct 30 22:58:20.023979 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:20.023933 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-l56rn_512e3b58-f29f-404d-8fef-4675058fac54/manager/0.log" Oct 30 22:58:20.255535 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:20.255504 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4xv26_0591cd90-06f6-40c3-a2bf-411dad660142/limitador/0.log" Oct 30 22:58:20.461662 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:20.461619 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z6cfl_a300142a-c951-45c6-832a-ec9949b961d5/manager/0.log" Oct 30 22:58:21.182407 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:21.182376 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2_af869660-acc6-4f23-a134-a9688b49b17b/istio-proxy/0.log" Oct 30 22:58:21.411914 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:21.411882 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c9rwj_8e65e971-7f9c-4007-822c-243c847822eb/discovery/0.log" Oct 30 22:58:22.068226 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:22.068186 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mgw8n_b6e11781-e771-4f4d-8458-3966ee881c40/istio-proxy/0.log" Oct 30 22:58:22.286737 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:22.286710 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm_3306eb24-d85b-4e7e-b4b4-e4f71c01050b/istio-proxy/0.log" Oct 30 22:58:22.958785 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:22.958753 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-755f8d7949-984mn_56e7aa71-e129-4ceb-91a9-06770acc40f7/router/0.log" Oct 30 22:58:24.183201 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:24.183171 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-kvcc8_29388185-34cf-4bb4-b6f3-331e792795c2/main/0.log" Oct 30 22:58:24.409076 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:24.409048 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-874n7_b6dfb179-54bc-4c3a-92ec-9bd5ab387460/main/0.log" Oct 30 22:58:25.117993 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:25.117934 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-qjj6n_0a896d62-f666-411c-9a73-e8bb1ac18267/main/0.log" Oct 30 22:58:25.375275 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:58:25.375173 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-jgrlq_27da8df3-0147-4496-9997-2e55b71fa988/main/0.log" Oct 30 22:58:39.876144 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:58:39.876109 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:59:06.791729 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:06.791699 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z258k_70fbe0ee-70e4-47b1-8094-4faa96292c6b/global-pull-secret-syncer/0.log" Oct 30 22:59:07.382300 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:07.382273 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dq5vk_5197a548-5e64-49a3-a5b4-421898abe7df/connection-checker/0.log" Oct 30 22:59:07.830672 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:07.830600 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-f47zh_aff7cebc-f18e-4b88-bb91-bec7ed9ab097/connection-checker/0.log" Oct 30 22:59:08.255105 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:08.255073 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qcvmd_78532fdc-d35e-4ec3-a48a-18fe0bbe8484/connection-checker/0.log" Oct 30 22:59:09.200259 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:09.200230 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lhsfr_40cea1c9-e71c-4451-926c-398f36e9691c/konnectivity-agent/0.log" Oct 30 22:59:09.883497 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:59:09.883459 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor" Oct 30 22:59:10.979711 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:10.979680 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-104.us-west-1.compute.internal_ff30ba81bde53b1c4da7b24cd2c2a4b9/haproxy/0.log" Oct 30 22:59:11.755727 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:11.755697 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lwspj_dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1/manager/0.log" Oct 30 22:59:11.915931 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:11.915903 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lwspj_dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1/manager/0.log" Oct 30 22:59:13.342601 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:13.342573 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lv5fm_36da7174-ce14-4f51-ae5e-21bd5b72fcae/server/0.log" Oct 30 22:59:13.598625 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:13.598551 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lv5fm_36da7174-ce14-4f51-ae5e-21bd5b72fcae/server/0.log" Oct 30 22:59:14.980487 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:14.980458 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qf8tr_a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda/proxy/0.log" Oct 30 22:59:15.251235 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:15.251161 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qf8tr_a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda/proxy/0.log" Oct 30 22:59:16.332118 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:16.332087 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77dd9fff5-h8rlb_2d74b772-c978-43f7-a84b-f23f68145de2/authorino/0.log" Oct 30 22:59:16.664437 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:16.664407 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-llb2k_019fe412-f0e3-48d4-9878-963b1a067f08/manager/0.log" Oct 30 22:59:17.091243 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.091216 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tmw28_067cbc96-3d21-422f-b69f-ffa11e3c5706/manager/0.log" Oct 30 22:59:17.278816 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.278785 4859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4"] Oct 30 22:59:17.279265 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.279249 4859 cpu_manager.go:414] "containerMap: removing container" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" Oct 30 22:59:17.279326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.279267 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" Oct 30 22:59:17.279326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.279288 4859 cpu_manager.go:414] "containerMap: removing container" podUID="e6a27b5c-b64d-40a6-be10-12011e09b3ee" containerName="revoke-keys" Oct 30 22:59:17.279326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.279293 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a27b5c-b64d-40a6-be10-12011e09b3ee" containerName="revoke-keys" Oct 30 22:59:17.279445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.279332 4859 cpu_manager.go:414] "containerMap: removing container" podUID="39d7d334-d9ba-4857-9161-3c5fc602194b" containerName="payload-processing" Oct 30 22:59:17.279445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.279338 4859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d7d334-d9ba-4857-9161-3c5fc602194b" containerName="payload-processing" Oct 30 22:59:17.279445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.279413 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d7d334-d9ba-4857-9161-3c5fc602194b" containerName="payload-processing" Oct 30 22:59:17.279445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.279425 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5de8966-9960-46fb-8704-ae704cc77a6e" containerName="main" Oct 30 22:59:17.279445 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.279434 4859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a27b5c-b64d-40a6-be10-12011e09b3ee" containerName="revoke-keys" Oct 30 22:59:17.282399 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.282382 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.322326 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.322301 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-proc\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.322444 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.322372 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-sys\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.322486 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.322459 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-podres\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.322575 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.322557 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7n6\" (UniqueName: \"kubernetes.io/projected/27cc0b94-f69f-422e-85c6-ee3362b28fad-kube-api-access-xs7n6\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.322695 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.322680 4859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-lib-modules\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.371957 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.371874 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4"] Oct 30 22:59:17.419353 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.419323 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-276rx\"/\"kube-root-ca.crt\"" Oct 30 22:59:17.423762 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.423734 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-lib-modules\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.423899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.423767 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-proc\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.423899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.423815 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-sys\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.423899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.423831 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-podres\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.423899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.423855 4859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7n6\" (UniqueName: \"kubernetes.io/projected/27cc0b94-f69f-422e-85c6-ee3362b28fad-kube-api-access-xs7n6\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.423899 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.423876 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-lib-modules\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.424153 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.423902 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-proc\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.424153 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.423911 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-sys\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.424153 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.423999 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27cc0b94-f69f-422e-85c6-ee3362b28fad-podres\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.444011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.443984 4859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-276rx\"/\"default-dockercfg-pbpfk\"" Oct 30 22:59:17.444011 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.443984 4859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-276rx\"/\"openshift-service-ca.crt\"" Oct 30 22:59:17.483298 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.483272 4859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7n6\" (UniqueName: \"kubernetes.io/projected/27cc0b94-f69f-422e-85c6-ee3362b28fad-kube-api-access-xs7n6\") pod \"perf-node-gather-daemonset-x5fw4\" (UID: \"27cc0b94-f69f-422e-85c6-ee3362b28fad\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.594274 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.594253 4859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:17.966157 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:17.966130 4859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4"] Oct 30 22:59:18.387753 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:18.387710 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" event={"ID":"27cc0b94-f69f-422e-85c6-ee3362b28fad","Type":"ContainerStarted","Data":"d09078271cb4a207914f13fe3436150185e1603aa09a1d5055866141e972c890"} Oct 30 22:59:18.388178 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:18.387798 4859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" event={"ID":"27cc0b94-f69f-422e-85c6-ee3362b28fad","Type":"ContainerStarted","Data":"075c4a899715b7c8fa3562d99b4b57d635ff214ce25a057964f23428c5b7f9ad"} Oct 30 22:59:18.388178 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:18.387838 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:18.512591 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:18.512541 4859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" podStartSLOduration=1.5125283139999999 podStartE2EDuration="1.512528314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:18.510477347 +0000 UTC m=+4615.883104091" watchObservedRunningTime="2026-10-30 22:59:18.512528314 +0000 UTC m=+4615.885155077" Oct 30 22:59:18.543131 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:18.543098 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-l56rn_512e3b58-f29f-404d-8fef-4675058fac54/manager/0.log" Oct 30 22:59:18.647228 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:18.647162 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lwspj_dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1/manager/0.log" Oct 30 22:59:18.700709 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:18.700683 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lwspj_dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1/manager/0.log" Oct 30 22:59:18.869430 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:18.869388 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4xv26_0591cd90-06f6-40c3-a2bf-411dad660142/limitador/0.log" Oct 30 22:59:19.326269 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:19.326241 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z6cfl_a300142a-c951-45c6-832a-ec9949b961d5/manager/0.log" Oct 30 22:59:20.537259 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:20.537233 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lv5fm_36da7174-ce14-4f51-ae5e-21bd5b72fcae/server/0.log" Oct 30 22:59:20.598935 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:20.598912 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lv5fm_36da7174-ce14-4f51-ae5e-21bd5b72fcae/server/0.log" Oct 30 22:59:22.266237 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:22.266195 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qf8tr_a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda/proxy/0.log" Oct 30 22:59:22.331645 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:22.331619 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qf8tr_a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda/proxy/0.log" Oct 30 22:59:24.403926 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:24.403896 4859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-x5fw4" Oct 30 22:59:25.023352 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:25.023319 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-xmt2j_3f03966a-97af-46f0-99a5-d50a4f54e744/cluster-monitoring-operator/0.log" Oct 30 22:59:25.439971 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:25.439933 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lwspj_dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1/manager/0.log" Oct 30 22:59:25.539151 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:25.539118 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lwspj_dc8d7267-ae9c-4a80-b53b-565ca8f2f8c1/manager/0.log" Oct 30 22:59:27.108832 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:27.108805 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lv5fm_36da7174-ce14-4f51-ae5e-21bd5b72fcae/server/0.log" Oct 30 22:59:27.250431 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:27.250409 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lv5fm_36da7174-ce14-4f51-ae5e-21bd5b72fcae/server/0.log" Oct 30 22:59:28.816746 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:28.816720 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qf8tr_a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda/proxy/0.log" Oct 30 22:59:28.870951 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:28.870923 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gczg9_2c573208-a84b-4845-bbea-640f0ee93603/node-exporter/0.log" Oct 30 22:59:28.962036 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:28.962007 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qf8tr_a8dc8a84-5fe7-4d5d-b9e1-7891be6fbeda/proxy/0.log" Oct 30 22:59:29.290268 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:29.290243 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gczg9_2c573208-a84b-4845-bbea-640f0ee93603/kube-rbac-proxy/0.log" Oct 30 22:59:29.727255 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:29.727228 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gczg9_2c573208-a84b-4845-bbea-640f0ee93603/init-textfile/0.log" Oct 30 22:59:32.391901 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:32.391873 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2_af869660-acc6-4f23-a134-a9688b49b17b/istio-proxy/0.log" Oct 30 22:59:32.546915 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:32.546885 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-ccbt2_af869660-acc6-4f23-a134-a9688b49b17b/istio-proxy/0.log" Oct 30 22:59:35.082611 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:35.082521 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c9rwj_8e65e971-7f9c-4007-822c-243c847822eb/discovery/0.log" Oct 30 22:59:35.335005 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:35.334900 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c9rwj_8e65e971-7f9c-4007-822c-243c847822eb/discovery/0.log" Oct 30 22:59:35.652685 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:35.652638 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mgw8n_b6e11781-e771-4f4d-8458-3966ee881c40/istio-proxy/0.log" Oct 30 22:59:35.866405 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:35.866376 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm_3306eb24-d85b-4e7e-b4b4-e4f71c01050b/istio-proxy/0.log" Oct 30 22:59:35.877054 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:35.877033 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mgw8n_b6e11781-e771-4f4d-8458-3966ee881c40/istio-proxy/0.log" Oct 30 22:59:36.091220 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:36.091184 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-z6kvm_3306eb24-d85b-4e7e-b4b4-e4f71c01050b/istio-proxy/0.log" Oct 30 22:59:36.510240 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:36.510168 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-755f8d7949-984mn_56e7aa71-e129-4ceb-91a9-06770acc40f7/router/0.log" Oct 30 22:59:36.743334 ip-10-0-141-104 kubenswrapper[4859]: I1030 22:59:36.743295 4859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-755f8d7949-984mn_56e7aa71-e129-4ceb-91a9-06770acc40f7/router/0.log" Oct 30 22:59:39.874668 ip-10-0-141-104 kubenswrapper[4859]: E1030 22:59:39.874628 4859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r246j" containerName="extractor"